Quick recipe for Layer2 Circuit local switching

I am always forgetting how to do l2circuits in Juniper, partially as there seem to be a zillion ways to configure encapsulation and VLAN handling, all of which seem to commit but seemingly very few seem to work.

This is a super quick note-to-self which describes how to locally switch (could simply be extended to LDP-signalled l2circuit over an MPLS core though) a point-to-point circuit, one end VLAN tagged and the other end untagged.

For this example, we have two interfaces – both on a single MX router called mx2.lab. Our ‘tagged’ or NNI facing interface is xe-0/0/1, and we’re using VLAN 250. Our ‘untagged’ or CPE facing interface is xe-2/2/1, not using a VLAN at all (dedicating the whole interface). This can (again) be expanded to use S/C tags, multiple encapsulations etc, but I’m not going there yet.

What we’re aiming to see is traffic coming in on a VLAN tagged interface and being locally switched to an untagged interface. To lab this, I have a VLAN-tagged interface with IP and an untagged VM, sitting on – when it’s configured, they should be able to ping one another.


We need 3 chunks of config to make this config work:

  • The tagged interface
set interfaces xe-0/0/1 unit 250 description "Tagged interface for L2Circuit Test"
set interfaces xe-0/0/1 unit 250 encapsulation vlan-ccc
set interfaces xe-0/0/1 unit 250 vlan-id 250
set interfaces xe-0/0/1 unit 250 family ccc
  • The untagged interface
set interfaces xe-2/2/1 description "Untagged interface for L2Circuit Test"
set interfaces xe-2/2/1 mtu 9100
set interfaces xe-2/2/1 encapsulation ethernet-ccc
set interfaces xe-2/2/1 unit 0 input-vlan-map push
set interfaces xe-2/2/1 unit 0 input-vlan-map vlan-id 250
set interfaces xe-2/2/1 unit 0 output-vlan-map pop
  • The l2circuit config
set protocols l2circuit local-switching interface xe-0/0/1.250 end-interface interface xe-2/2/1.0
set protocols l2circuit local-switching interface xe-0/0/1.250 ignore-encapsulation-mismatch
set protocols l2circuit local-switching interface xe-0/0/1.250 ignore-mtu-mismatch

With that config loaded on mx2.lab, packets will fly between the untagged VM on xe-2/2/1 and xe-0/0/1.250


How to connect your Raspberry Pi to eduroam

Note – I took much of the code snippet here from ‘Sruc‘ on the RPI forums, but wanted to post a clear method that I know works. Cheers Sruc!

The eduroam network (for universities, researchers and highschools around the world) is a great thing. One login lets you connect to wifi access points all over the place, as long as you’re enrolled in or working for a participating organisation.

One thing that bugged me out of the box with the Raspberry Pi (in my case, a Raspberry Pi 3 running Pixel) – was the Enterprise WPA wifi not working out of the box.

Follow these simple steps to get it working:

  • Open a Terminal from your Pi’s gui (or just use the shell if you don’t have a gui!)
  • Open up the wpa_supplicant.conf file:
sudo nano /etc/wpa_supplicant/wpa_supplicant.conf
  • Paste in the following, changing the bits you normally use to log in to eduroam (your university/whatever email and password is normally what you use for authentication)

(Add this snippet below what’s already in the file, change the ‘identity’ and ‘password’ fields!)

  • Save and exit the editor (in nano that’s CTRL-O, Enter, CTRL-X)
  • Now we need to tell the Pi to reload the file, again, in the Terminal or shell
sudo wpa_supplicant -i wlan0 -c /etc/wpa_supplicant/wpa_supplicant.conf
  • I find a reboot here is necessary, so flip the Pi and wait for it to boot. When it returns, you should be connected to eduroam (as long as your Pi can see the eduroam SSID!)

Note – I am not sure if this will work for all instances of eduroam, as some Universities etc handle authentication differently – please check your organisation’s help pages or get in touch with them first – they usually have a guide.

Very useful Ubuntu 16 Networking Note

I hate when things change for no good reason. This week, it’s the interface naming of ethernet on Ubuntu 16. No more does it default to ‘eth0’.. It uses some other ‘ens’ style.. Garbage!

First up, find your ethernet interfaces (this VM has 1 interface to start):

dmesg | grep -i eth
james@GREG:~$ dmesg | grep -i eth
[ 1.479809] vmxnet3 0000:03:00.0 ens33: NIC Link is Up 10000 Mbps

Bah, looks gross!

Fix it by editing your grub config:

sudo nano /etc/default/grub

Change the line GRUB_CMDLINE_LINUX=””

to  GRUB_CMDLINE_LINUX=”net.ifnames=0 biosdevname=0″

Regenerate your grub file:

sudo grub-mkconfig -o /boot/grub/grub.cfg

Edit your /etc/network/interfaces file, change the names to eth0, eth1 etc

# The loopback network interface
auto lo
iface lo inet loopback

# The primary network interface
auto eth0
iface eth0 inet static

Reboot, and voila.

If you add a new interface, it will come on as ethX, following the already provisioned interfaces.

Now it looks better (I added a new 10G interface, and it comes in as eth1)

james@GREG:~$ dmesg | grep -i eth
[    1.479809] vmxnet3 0000:03:00.0 eth0: NIC Link is Up 10000 Mbps
[    2.498115] vmxnet3 0000:0b:00.0 eth1: NIC Link is Up 10000 Mbps

Awww yeeeeah

Openstack Newton – Provider Network Issue

When playing with an Openstack POC recently, I nearly pulled my hair out. I am running a flat provider network between my compute nodes (all Ubuntu 16.04), which connect via  Cisco 2900 with an inbuilt switch module. The Cisco has gateway addresses for the dual-stack host networks. I was using native IPv6 and NAT’d private space for IPv4.

Whenever I went to launch an instance, DHCP would work (SLAAC for v6), and the Horizon front-end would show the generated addresses assigned to the instance. Looks good. Going into the console of the instance, I’d see (with ifconfig) no IP addresses on my host NIC.. Looking in the “neutron-dhcp-agent.log” log, I would see:

2017-03-24 23:49:44.169 2476 WARNING stevedore.named [req-6710e8a6-5991-446e-b8d2-5af6c9d27625 – – – – -] Could not load neutron.agent.linux.interface.BridgeInterfaceDriver

Whenever an instance was on. Cycling over and over for the number of instances. When I looked at the bridge status (in my topology the eno4 phyical interface of the compute/controller nodes are connected to the physical provider network), I would not see eno4 in the bridge created to connect hosts to physical. The way Openstack Neutron does this is to build a bridge, then add the physical and TAP interfaces). Mine was missing. Why…

Turns out – I had an IPv6 DHCP scope on my Cisco provider network interface facing the Openstack environment. As soon as I removed this piece of config (and simply left the IPv4 and IPv6 gateways on that interface) – eno4 showed up in the bridge and it all went smoothly.

What a mission.

Xiaomi Mi 5s – Overcoming a fake ROM


I bought a fancy pants new Xiaomi phone. My Samsung S5 was getting a bit long in the tooth. I wanted another flagship, but Samsung’s ROMs have been bloating up recently and while I’m a custom ROM fanboy, I was getting sick of Cyanogenmod and the occasional app not working with an unlocked bootloader, custom ROM etc.

So I got a Xiaomi Mi 5S for about $400 NZD from Banggood. The phone and case arrived 7 days after I ordered, so far so good. What I didn’t know in advance was Banggood (and other vendors) sneakily put their own ROM on the phone which is rambo jambo’d with Malware, Adware and can’t receive OTA (over the air) software updates – in short, it’s a shit sandwich.

The MIUI forums are a great place to get help, but if you have a Banggood Mi 5s phone with an 8.x.x.x fake ROM – you can probably follow these steps.

  1. First up, confirm you are on a fake ROM. This page is a good way to do so. I had and it looked bad news. I couldn’t do a software update or run Pokemon Go, so I decided I was affected and moved on.
  2. Unlock your Boot Loader. What does this mean? Well, it’s a setting on an Android phone that lets you install custom software. In our case, we’re actually trying to do the opposite, by rolling this dodgy fake ROM back to the official MIUI release. Nevertheless, we need to unlock our BL. Xiaomi makes us do this via logging in to their website and requesting an unlock on our Mi account. Simply create a Mi account and request an unlock. It might take a few days – mine took approx. 72 hours to have a verification SMS.
  3. Once we have the confirmation that the Mi account is enabled for unlocking, we need to download some software (unfortunately you need a Windows PC for this – although I did succeed using a VMWare Fusion VM on OSX). This software is the Unlocking app – we need to log in to the Mi account on the phone and inside this programme simultaneously – otherwise it will stall on 50% in the unlocking process. So – click here to get the latest Mi Flash.
  4. Open Mi Flash programme, turn your phone on in Recovery mode (Hold volume Up/Pwr button at the same time, from when the phone is off). Log in on the application and select Unlock.. This should work smoothly.
  5. Now, we have unlocked the bootloader, we should be able to use the Update application on the phone itself (usually in a folder called Tools). Best bet is to go to the official source and grab a Global Rom (assuming you don’t want a Chinese one) – over here. Once you have the rom, put it on your phone’s internal storage in a folder called downloaded_rom (create one if it doesn’t exist), open the aforementioned Update app and select the .zip file you just uploaded. It will take about 45 mins to work.

Mean buzz – you have a official Xiaomi phone/ROM – not some dodgy crippleware. Instead of asking questions here, I recommend heading to the MIUI forums, already linked. Try and only use official software from Xiaomi/MIUI.. Not everyone out there is friendly.


Source Specific Multicast with iperf

oooh iperf ssm

As part of my lab at work, I need to have lots of traffic flying around all over the place. Unicast IPv4 and IPv6, of course – but also traffic in L3VPNs and multicast traffic. Multicast is a big part of the day-to-day traffic load in my production network, so it’s important to be there in the lab too.

I’ve used a variety of tools to generate multicast traffic in the past, more often than not the excellent OMPing. This time, however, I wanted to really chuck some volume around, to make my stats look nice and to really show up on NFSen. iperf is the natural choice for generating lots of traffic in a hurry, I’m using iperf3 to throw about 20Gbit/s around the lab core, but for some reason the developers have removed multicast functionality from the new and improved iperf3.

ASM (or any-source multicast) is (more) traditional multicast. It relies on interested parties joining a group (a multicast address) where any sender can be considered the source (*,G). It works well when your environment is well set up, with a rendezvous point and protocols such as PIM-SM (and MSDP when you wanna go multi-domain). Iperf handles this pretty easily, simply set up a server to ‘listen’ on one end, and a client to send on the other. This can be achieved by having two hosts, connected via a router which acts as the PIM-SM rendezvous point and has IGMPv2 on the interfaces, so the listener can tell it’s router that it’s keen on having a listen.

On the listener:
iperf -s -B -u -f m -i 1

On the sender:
iperf -c -u -b 100m -f m -i 1 -t -1 -T 32

On the listener, we’re telling iperf to be a server (-s), listening to an ASM multicast group (-B), using UDP (-u), formatting the output in megabit/s (-f) and telling us what’s up every 1 second (-i).

On the sender, we’re telling iperf to be a client (-c), using UDP, sending a 100Mbit/s stream (-b), sending forever (-t) and setting the multicast TTL to 32 (-T) – overkill, but hey.

This is a quick and easy way to check a basic ASM setup. I use it to confirm my multi-domain lab setup is working with multicast as I would expect and to generate lots of traffic to amuse myself.

Getting source specific

Source specific multicast is a bit cooler than ASM, because it should work for more people, more easily. If you have a single source (like a TV encoder, or some kind of unique data source) that you want plenty of receivers to see, and you have a nice way of telling them about your source address (like a higher-layer application – or in our case, manual config) – then SSM is the protocol for you.

To make SSM work, you really only need IGMPv3 in your network. Most *nix OS’s and even Windows supports IGMPv3 – usually by default. To check on your *nix host, you can run:
cat /proc/sys/net/ipv4/conf/eth0/force_igmp_version

If that returns a 0, and you’re on a modern-ish OS, you’re good to go. You can force your kernel to use IGMPv3 per interface, with:
echo "3" > /proc/sys/net/ipv4/conf/eth0/force_igmp_version

While most OS’s support IGMPv3 out of the box, plenty of network administrators in your friendly local LAN have probably forgotten to turn it on, or have left it with IGMPv2 and called it a day. No good.

Assuming you’re all good with IGMPv3, you then need to have an application which can listen not only for a particular multicast group address, but also a specific source. iperf, by default, doesn’t support this. Lucky for us, then, noushi has written an extension to iperf that allows for a couple of new flags, to set a specific source and interface. You can grab it here.

If you already had iperf installed, remove it. In Debian/Ubuntu, that would be:
sudo apt-get autoremove iperf -y

Unzip the file from git, then we compile:
sudo make install

Now we can run a client to send multicast as before (but this time, sending to the SSM range):
iperf -c -u -b 10m -f m -i 1 -t -1 -T 32 -p 5002

I’m running it on a different port (-p) from default, because I want SSM and ASM at the same time.

For the listener, we need to tell our newly improved iperf to listen for a particular source (the regular old IP address of our sender):
iperf -s -B -u -f m -i 1 -p 5002 -O -X eth0.101

The two new flags are there, along with our SSM multicast group (and new port). The -O flag sets the SSM source and -X tells iperf to use a particular interface. I’m not sure -X is new, but I’ve never used it before – so let’s say it is.

If it’s working, we’ll see a 10Mbit/s SSM stream turn up on the listener.

nfsen + debian + apache = d’oh

I was re-doing one of my lab monitoring tools, a VM that hosted too many sparse and poorly maintained pieces of software. Now re-homing each bit onto its own VM (partially for sanity) – I ended up re-installing the excellent NFSen (a netflow monitoring tool/frontend for nfdump).

The software includes a directory named ‘icons’ in the web root, which doesn’t seem insane to me. What is insane, however, is Apache’s decision (by default!) to include an alias for a folder named ‘icons’ in the root. That means that without knowing it, the NFSen icons folder was being redirected to /usr/share/apache2/…/ whatever. That caused a headache.

To find this out, I ran:
cd /etc/apache2
grep -iR /usr/share *

This told me about the dang alias file, /etc/apache2/mods-available/alias.conf

I went into that file, commented out this dumb default, reset apache and now it’s away laughing.

QoS for your Linux Plex box

FireQos in action
FireQos in action

When Jim Salter posted about FireQos the other day, it made me take note. FireQos is a unix’y firewall ‘for humans’. In my day job, QoS is a complex and multi-faceted thing, requiring tonnes of design, thought and understanding to implement correctly (not to mention hardware). It has dramatic effects on network traffic when set up correctly, but that usually means end-to-end config across a domain, so marking at one end of the network translates to actions all the way through. That’s a bit much for home.

I was interested, as I had a problem. Behind my TV I have an Intel NUC, with a little i3 processor and 802.11n wifi. I use it to torrent things, run a Plex server and be a multi-purpose Linux machine for my own needs the rest of the time. (OwnCloud is still running on the Raspberry Pi 3, mind you). When I was pulling down a delicious new Debian image at 12MB/s, and trying to watch something on Plex (via the PS4), things got a bit choppy. Try to VNC into the box from my laptop to throttle the torrent was always annoying, it could take minutes for the screen to refresh if a very hearty download was going on. Like most nerds, the slightest delay caused by my own setup was slowly tearing me apart.

This is where FireQos comes in. With a very simple install and a couple of minutes of settings out of the way, the performance improved dramatically. All I did was prioritise the traffic for Plex, SSH, VNC and browsing over torrents/anything else – and like magic, everything works smoothly altogether – with no throttling on the torrent client.

Remember before where I said QoS really needs to be end-to-end in the network to make a difference? In this case, not true. By simply tweaking the Linux handling of packets, things have gotten much better with the rest of the network unaware anything is happening. Obviously, this would improve if I had a router that was also participating in the fun, but I don’t.. Yet. At the moment, if another device tries to use the network when a full torrent storm is going on, it’s toast.

Anyhow, check out the FireQos tutorial here, and give it a crack yourself. There’s basically no risk, go nuts.

Here’s my fireqos.conf file, so you can copypasta it if you like.


interface $DEVICE world-in input rate $INPUT_SPEED
interface $DEVICE world-out output rate $OUTPUT_SPEED

interface $DEVICE world-in input rate $INPUT_SPEED
   class interactive commit 20%
 	match udp port 53         
    	match tcp port 22             
    	match icmp                    
    	match tcp sports 5222,5228    
    	match tcp sports 5223

    class plex commit 50%
    	match udp port 1900
    	match tcp port 3005
    	match udp port 5353
    	match tcp port 8324
    	match udp port 32410
    	match udp port 32412
    	match udp port 32413
    	match udp port 32414
    	match tcp port 32469

    class vnc commit 5000kbit
    	match tcp port 5901

   class surfing commit 20%
	match tcp sports 0:1023

   class synacks            
      match tcp syn                    
      match tcp ack                 

   class default

   class torrents
      match dports 6881:6999
      match dport 51414 prio 1 

interface $DEVICE world-out output rate $OUTPUT_SPEED
   class interactive commit 20%
      match udp port 53             
      match tcp port 22             
      match icmp                    
      match tcp dports 5222,5228    
      match tcp dports 5223         

    class plex commit 50%
      match udp port 1900
      match tcp port 3005
      match udp port 5353
      match tcp port 8324
      match udp port 32410
      match udp port 32412
      match udp port 32413
      match udp port 32414
      match tcp port 32469

    class vnc commit 5000kbit
      match tcp port 5901

   class surfing commit 20%
      match tcp dports 0:1023

   class synacks                       
      match tcp syn                    
      match tcp ack                   

   class default

   class torrents
      match dports 6881:6999        
      match dport 51414 prio 1

Best Windows TFTP Client

This is mostly a note to self.

3CDaemon was always my Windows TFTP server of choice, but finding a valid .exe that works on Windows 7 64bit and isn’t riddled with viruses is a problem when you’re in a hurry. Ph Jounin has written the lovely Tftpd64. You can get it from the main site here (and in 32bit flavours if you like). I’ll upload the zip here too, incase it ever goes offline.

I don’t mind if you never want to trust a download from a random blog, but for my own stress levels, now I have it in the cloud 🙂


Upgrading Junos issue – not enough space

Quick note, mostly for my own reference down the track.

I have in the lab a MPC3E-NG FQ and a MIC3-100G-DWDM card. To those of you not Juniper ensconced, that’s a chassis-wide slot and a 100Gbit/s DWDM (tunable optic) card. Wowee, I hear you say. Anyway, the 100G DWDM card requires a fairly spicy new version of Junos, one with a new underlying FreeBSD core at the heart. My lab was running on Junos 14.1R5.5, an already pretty recent version – but for 100G across my lab I need to use the DWDM card, inconjunction with some Infinera DWDM kit for good measure.

Normally, a Junos upgrade is fairly painless. In this case however, I was getting errors that I couldn’t understand. Here is how I pushed past them.

When going from 14.x to 15.1F6, Juniper don’t recommend running validation of the software release against your config.

james@mxZ.lab.re1> request system software add /var/tmp/junos-install-mx-x86-64-15.1F6.9.tgz no-validate re1

When I ran this on one RE (RE0) it went through just fine. On RE1 however, I got this:

Installing package ‘/var/tmp/junos-install-mx-x86-64-15.1F6.9.tgz’ …
Verified manifest signed by PackageProductionEc_2016
Verified manifest signed by PackageProductionRSA_2016
Verified manifest signed by PackageProductionRSA_2016
Verified contents.iso
Verified issu-indb.tgz
Verified junos-x86-64.tgz
Verified kernel
Verified metatags
Verified package.xml
Verified pkgtools.tgz
camcontrol: not found
camcontrol: not found
camcontrol: not found
Verified manifest signed by PackageProductionEc_2016
Saving the config files …
NOTICE: uncommitted changes have been saved in /var/db/config/juniper.conf.pre-install
tar: contents/jpfe-wrlinux.tgz: Wrote only 0 of 10240 bytes
tar: Skipping to next header
tar: Archive contains obsolescent base-64 headers
tar: contents/jplatform-ex92xx.tgz: Wrote only 0 of 10240 bytes


tar: Skipping to next header
tar: Error exit delayed from previous errors
ERROR: junos-install fails post-install

This looks a bit like something has gone wrong, but it’s not immediately obvious what it is. I am using a .tgz of Junos 15 that is sitting on my RE1’s /var partition, and it has 2.1 GB hard-drive space free (show system storage)… Queue a few hours of headscratching.

Turns out, my assumption on how much space is actually required when upgrading from Junos 14 to 15 was completely under. I started with about 2GB free, but when I was finally successful I had 11GB free. Once the install was complete, I was down to 7.7G, which means the installation process uses up 3.3GB all by itself. I guess that’s not crazy, as the tarball is 1.9GB to begin with, but the error output didn’t make it clear enough for me, a stupid man.

Here is how I overcame the obvious and got Junos 15.1F6 installed 🙂

1: Jump into shell

james@mxZ.lab.re1> start shell

2: Become root

james@mxZ.lab.re1> su
<enter password>

3: Find all the files that are > 500MB. We’re looking for things in /var

root@mxZ% find / -size +500000

4: Delete any of the old .tgz files from previous releases. If you want to keep them, SCP them off first.

root@mxZ% rm /var/sw/pkg/jinstall64-12.1R2.9-domestic-signed.tgz

etc etc

5: Check you now have > 3.5 GB free. Or go all out like me and have 11GB free, whatever.

6: Upgrade Junos and get back to work!