EVE-NG rules. As far as network simulation software goes, it’s the best.
When studying or otherwise, EVE-NG is the way I prefer to try things out. One thing that happens, however, when using virtualised networks, is you obscure some underlaying things – one of them being MTU. In a previous post, I went through how the base OS that EVE-NG runs on virtualises the links between routers and switches, here I will show a way to boost the MTU these virtual network links use, so that we can throw proper jumbos across the network.
In this topology, I have 2 routers, connected with dual Ethernet links, configured in a LAG. This doesn’t affect MTU at all, I just thought I’d mention it so it’s not confusing.
The link between these routers (ae0) is set to a layer-2 MTU of 9192, which is the maximum for the platform (Juniper vSRX 3.0). This means that we should be able to send an IP packet (like a ping) of over 9000 bytes.. And yet – we can’t:
root@R1> ping 22.214.171.124 size 9001 do-not-fragment PING 126.96.36.199 (188.8.131.52): 9001 data bytes --- 184.108.40.206 ping statistics --- 4 packets transmitted, 0 packets received, 100% packet loss
So – let’s first see why, then fix it.
If we SSH onto our EVE-NG server, and take a look at the virtual network that makes up the link between ge-0/0/1 on R1 and ge-0/0/1 on R2, as well as ge-0/0/2 on both as well (since we’re in a 2 member LAG). See previous post for more detail on how to track down which link is which and what they’re called inside EVE-NG’s OS:
root@eve-ng-2:~# ifconfig vunl0_26_2 vunl0_26_2 Link encap:Ethernet HWaddr d6:41:90:79:cb:60 UP BROADCAST RUNNING MULTICAST MTU:9000 Metric:1 RX packets:405 errors:0 dropped:0 overruns:0 frame:0 TX packets:868 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:66286 (66.2 KB) TX bytes:88080 (88.0 KB)
There it is. At layer-2, we have MTU 9000. As the virtual routers are assuming a piece of wire with no MTU at all connecting them to the other side, there is a problem. I think I’m MTU 9126, but in reality I’m down to 9000 on the wire. Which explains why I can send a ping that’s nearly 9000 bytes:
root@R1> ping 220.127.116.11 size 8976 do-not-fragment PING 18.104.22.168 (22.214.171.124): 8976 data bytes 8984 bytes from 126.96.36.199: icmp_seq=0 ttl=64 time=1.150 ms 8984 bytes from 188.8.131.52: icmp_seq=1 ttl=64 time=1.028 ms 8984 bytes from 184.108.40.206: icmp_seq=2 ttl=64 time=1.066 ms 8984 bytes from 220.127.116.11: icmp_seq=3 ttl=64 time=1.073 ms
Oh yeah. 8976 bytes at layer 3, plus IP and ethernet headers = 9000.
So – let’s isolate the 4 network interfaces that EVE-NG has generated to be my ge-0/0/1’s and ge-0/0/2’s, then set the MTUs to something high – like 9200:
root@eve-ng-2:~# ifconfig vunl0_26_2 mtu 9200 root@eve-ng-2:~# ifconfig vunl0_26_3 mtu 9200 root@eve-ng-2:~# ifconfig vunl0_27_2 mtu 9200 root@eve-ng-2:~# ifconfig vunl0_27_3 mtu 9200
When EVE-NG creates a ‘wire’ between two network devices, it really creates 2 virtual network interfaces and assigns each of them to a Linux bridge. In my case, with 4 total interfaces, I have two bridges. They inherit the MTU of the smallest member, so both of my bridges now should be 9200:
root@eve-ng-2:~# ifconfig vnet0_37 vnet0_37 Link encap:Ethernet HWaddr 0a:c8:48:ad:34:71 UP BROADCAST RUNNING MULTICAST MTU:9200 Metric:1 RX packets:30596 errors:0 dropped:28163 overruns:0 frame:0 TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:3289806 (3.2 MB) TX bytes:0 (0.0 B)
Alright, we can see MTU on one of the bridges is now 9200 (it was 9000 before as both contributing interfaces were 9000) and we can also see a lot of dropped packets (because all my test pings at above 8976 bytes were crashing on the digital rocks).
So.. Can we send big pings now? Man. I hope so.
root@R1> ping 18.104.22.168 size 9100 do-not-fragment PING 22.214.171.124 (126.96.36.199): 9100 data bytes 9108 bytes from 188.8.131.52: icmp_seq=0 ttl=64 time=1.374 ms 9108 bytes from 184.108.40.206: icmp_seq=1 ttl=64 time=0.764 ms 9108 bytes from 220.127.116.11: icmp_seq=2 ttl=64 time=0.910 ms 9108 bytes from 18.104.22.168: icmp_seq=3 ttl=64 time=1.022 ms
Oh yes we can.
Note – this is not a permanent fix – it will last only as long as the virtual interfaces (or a server reboot) – and you need to do it for every link that is created. Perhaps it’s something to submit as a feature request to the EVE-NG team – but for now this will let us live in jumbo-harmony. Network links are ephemeral in EVE-NG, so statically setting this with a config file isn’t a great solution – but on the other hand we don’t drop our labs all that often – do we?