How to migrate a Raspberry Pi SD card to a bigger SD Card

Snazzy title, James.

I recently bought a 200GB SD card from Amazon for what I consider to be a completely crazy price (around $80USD). My Raspberry Pi 3 is running OwnCloud (howto post coming soon) and the 32GB card currently serving as the OS’ home is a little on the small side. It’s also quite an old card, so I’m worried it will one day up and die.

Here’s a quick step-by-step on how to move your SD card to a bigger one, using a Mac with OSX El Capitan (or basically any Unix based computer with an SD card reader).

Step 0: Backup anything important. Things can go wrong any time you mess with this stuff, so caution.

Step 1: Shutdown your Pi (pull the plug out)

Step 2: Take the SD card you want to copy out of the pi, and stick it in your Mac (with an adaptor). It will mount automatically, so run Disk Utility and unmount the ‘boot’ partition, which is what Raspbian OS calls it by default. Once the partitions are greyed out, they’re not mounted anymore.

Screen Shot 2016-04-26 at 2.28.51 pm

Step 3: Check you have space. You will need as much free hard-drive space on your Mac as equals the size of the SD card, even if it’s not full. Mine is 32GB, so I need 32GB hard drive space free. Sadly, I don’t have that much free on my tiny SSD, so I will be using an external USB 3 drive with ample space.

Step 4: Find the BSD name of your SD card reader/card. You do this by opening a terminal and entering:

Jamess-MacBook-Pro:~ james$ diskutil list

Yeah which will spit out a list of your connected media. We’re looking for something that matches the 32GB size of the SD Card (yours might be 8GB, 16GB… whatever)

/dev/disk2
   #:                       TYPE NAME                    SIZE       IDENTIFIER
   0:     FDisk_partition_scheme                        *31.4 GB    disk2

Here we see ‘disk2’ under Identifier, which is what we want.

Step 5: Find out where the file copy of the SD card should go. I want to stick it on my external HDD called SUGAR ROSS (for some reason), so I need to find where it’s mounted.

Jamess-MacBook-Pro:~ james$ cd /Volumes/
Jamess-MacBook-Pro:Volumes james$ ls
BOOTCAMP    Macintosh HD    SUGAR ROSS
Jamess-MacBook-Pro:Volumes james$ cd SUGAR\ ROSS/
Jamess-MacBook-Pro:SUGAR ROSS james$ pwd
/Volumes/SUGAR ROSS

There I show you that the 1TB external drive I have connected is located at /Volumes/Sugar\ Ross/ . The extra \ I have in there is an escape character, which helps Unix based Operating Systems handle things like spaces in folder names, which it didn’t historically like.

Step 6: Copy that floppy. We’re now going to use one of the oldest Unix commands there is – dd. Once again, in the terminal, tell the computer to make a direct byte-for-byte copy of the SD card and stick it in a file called sdcard.img (or whatever you fancy).

Jamess-MacBook-Pro:~ james$ sudo dd if=/dev/rdisk2 of=/Volumes/SUGAR\ ROSS/sdcard.img bs=1m

This will take a long while. It’s pretty oldschool and takes its job very seriously. You can tell what’s going on behind the scenes by pressing control-t every now and then. So far, mine has written 23GB in 1226 seconds, and still running. Won’t be much longer now, have a cup of tea. (I will not be doing this for my 200GB card.. Hopefully).

Step 7: Check your image. Ok, 27 minutes elapsed time later, and dd tells me it has finished. As I never believe anything my computer tells me, I want to check the output file.

Jamess-MacBook-Pro:SUGAR ROSS james$ cd /Volumes/SUGAR\ ROSS/
Jamess-MacBook-Pro:SUGAR ROSS james$ ls -la
total 61315256
d.........  1 james  staff         4096 21 Jan 19:19 $RECYCLE.BIN
d.........  1 james  staff         4096 26 Apr 14:22 .
d.........@ 5 root   admin          170 26 Apr 14:08 ..
-.........  1 james  staff  31393316864 26 Apr 14:49 sdcard.img

Using ‘ls -la’ shows me all the files on SUGAR ROSS, and I can see my nice .img file there at just shy of 32GB.

Step 8: Format wars. The SD card I am replacing my Pi’s with is 200GB, and I’m told that gargantuan SD cards have a funny filesystem that’s not compatible with the Pi. As I am writing a physical copy of the existing card onto the new one, I don’t think I’ll have a problem. If you do, I suggest formatting it to FAT32 and going from there. The dd process (this time in reverse) we just ran will take care of everything, as we didn’t copy a partition of the SD card, we copied the whole thing.

Step 9: Prepare the new card. Stick the new (bigger) SD Card into the Mac’s card reader. Use diskutil list again to find the BSD name of the disk, and then unmount any partitions that automatically mounted.

Step 10: Write to disk. This time, we use dd in reverse (and wait a lot longer).

Jamess-MacBook-Pro:~ james$ sudo dd if=/Volumes/SUGAR\ ROSS/sdcard.img of=/dev/disk2 bs=4m

Hint: make sure you unmount, not eject the SD card partition on the card you want to write out to. Disk Utility is good for this.

Step 11: Fire it up. Now that we’ve spent 11,000 seconds writing the 32GB image back to the new card, unmount it (if you succeed with dd in step 10, it should auto mount on OSX). Place the new card into the Pi and power it up. With any luck, it will fire up as it did before. Note, you’ll still only see the original filesize, for now..

pi@cloud:~ $ df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/root        29G  9.7G   18G  36% /
devtmpfs        483M     0  483M   0% /dev

(See how it still says 29G? That’s a good indication my old SD card (and all the partitions on it) were copied over to the 200G card.

Step 12: Expand the filesystem. Run the following command to stretch out the filesystem to the full 200G.

pi@cloud:~ $ sudo raspi-config

You can then select ‘Expand filesystem’ from the menu and reboot..

Step 13: Revel in your success.

pi@cloud:~ $ df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/root       181G  9.7G  164G   6% /
devtmpfs        483M     0  483M   0% /dev

Here we see 181GiB of usable space (roughly 200GB). Hooray!

Regex for private-ASN on Junos

Hi, super quick note to share something I’ve made to match 2 Byte Private ASN numbers (64512-65535) in Junos. You can apply it to match a community, or in an AS-PATH.

^((6451[2-9]|645[2-9][0-9]|64[6-9][0-9][0-9]|65[0-4][0-9][0-9]|655[0-2][0-9]|6553[0-5]).*)$

I will try and expand it to catch “0”, and also 4 Byte Private ASN, but I can’t burn time on that at the moment.

This one catches “0” (also an invalid community), but doesn’t work in Juniper as the \D toggle doesn’t compute…

^((6451[2-9]|645[2-9][0-9]|64[6-9][0-9][0-9]|65[0-4][0-9][0-9]|655[0-2][0-9]|6553[0-5]).*)|(\D0:).*$

RYU Faucet (SDN Controller) on Raspberry Pi

I think I finally have a use for one of my 3 Raspberry Pi computers – an SDN controller.

I have an Allied Telesis switch about to arrive, which I will be using as an SDN device – buzzy and buzzwordy at the same time. I was planning on running Ryu-Faucet as my controller stack on a VM, but I had the genius move of moving my R730 server into another location – thinking the SDN controller should be near to the switch for noob-level troubleshooting (despite that being kind of against the spirit of SDN…) – I went with the nextmost NIX thing I could find – a Raspberry Pi 2 running Ubuntu 14.04 LTS.

So – how to go about doing this from a fresh Ubuntu install? Some garbled notes to follow – but first up – I am not a software developer, I can barely code to save myself – so using things like PyPi are new to me.

First up – an apt-get update / upgrade (I run everything from root for this process)

james@ubuntu:/# sudo su
root@ubuntu:/home/ubuntu# apt-get update
root@ubuntu:/home/ubuntu# apt-get upgrade

That gets us up to date. On my home connection with the Raspberry Pi 2 it took about 15 minutes to complete. Next up, installing enough Python tools to install more Python tools.. Ubuntu 14.04 LTS comes with Python 2.7 by default – so that’s a good start.

Now, we grab Pip (more deets here):

root@ubuntu:/home/ubuntu# wget https://bootstrap.pypa.io/get-pip.py
root@ubuntu:/home/ubuntu# python get-pip.py

Cool, so that grabs Pip for us. Note: I ran get-pip.py twice, the first time it installed an old version, then it smoothly upgraded it (from version 1.5 to version 7.2!).

With Pip installed, we’re good to go installing Ryu-Faucet.. Except not quite. The instructions here don’t account for the fact that we’re not hardened Python devs (and that we’re running on an ARM CPU not an Intel one – I guess).

Grab some extra tools from apt:
root@ubuntu:/home/ubuntu# apt-get install python-dev
Now we can run Pip and pull down the file we want to install (ryu-faucet!):

root@ubuntu:/home/ubuntu# pip install https://pypi.python.org/packages/source/r/ryu-faucet/ryu-faucet-0.28a.tar.gz

Watch the magic occur!

root@ubuntu:/home/ubuntu# pip show ryu-faucet
---
Metadata-Version: 2.0
Name: ryu-faucet
Version: 0.28a0
Summary: Ryu application to perform Layer 2 switching with VLANs.
Home-page: http://http://openflowsdn.github.io/faucet
Author: Shivaram Mysore, OpenflowSDN.Org
Author-email: faucet-dev@OpenflowSDN.Org
License: Apache License 2.0
Location: /usr/local/lib/python2.7/dist-packages
Requires: ryu, pyyaml

Cool. Next time, we’ll configure the Pi’s single ethernet interface to handle management and SDN controller duties and fling some packets across the data plane.

Select an IPv6 address with a double-click in PuTTY

This ranks highly on the nerd-alert scale, but anyway, here we are.

Putty has a nice feature where you can select a word with a double click. It’s really handy, especially when you are dealing with IPv4 prefixes. It doesn’t work with IPv6 addresses by default. You can edit what Putty uses as a delimiter, under Settings -> Selection.

putty

Scroll down to ‘:’ (or whatever character you want to change the behaviour of!) and change its class from 1 to 2. If you save this as your default settings, from now on you can get a nice double click selection of IPv6 addresses.

More info on classes here.

FreeRADIUS 3 on Ubuntu 14.04 – some notes

Ugh. Nothing worse than setting out to install something relatively simple (like FreeRADIUS) only to find the documentation confusing, inconsistent and contradictory. FreeRADIUS has some pretty decent documentation for the actual operation of the software, but getting things going up-front can be a real mission, especially, if, like me, you’ve never done it before. I’m writing this as I go. My goal is to have a FreeRADIUS server running on my server, which I can use to test RADIUS functionality with some lab routers – Juniper MX480, MX960 and M120s.

First of all, why are you installing FreeRADIUS? What do you stand to gain here? If you already know and you’re hitting some glitches with the installation, these notes might help. If you want something a bit more comprehensive written by an expert – try here.

As FreeRADIUS comes with a pre-compiled package for my poorly chosen operating system (after many problems with Ubuntu 14.04 I’ve gone back to Debian 7.8 for nearly everything). That’s great news – simply run the lazy person’s favourite command:

sudo apt-get install freeradius -y

Now that bit is done, the docs tell you to run the ‘radiusd -X’ commands. From what I can see, that program doesn’t come with the Ubuntu package anymore. Simply replace it with ‘freeradius -X’ – it starts FreeRADIUS in debug mode, and away you go. Note – you might find that the ‘freeradius’ service has already started post installation.. Check with:

ps aux | grep freerad

If it’s in there, do a quick ‘sudo service freeradius stop’ and then you can fire it up again in debug (-X) mode.

Due to the way FreeRADIUS is recommended to be installed, the folder ‘/etc/freeradius’ (where the installer assumes you want things put) is owned by root. It might pay to do the rest of the work as root, if you can. Makes things a bit smoother, than having to use sudo all the time.

Add a user in the /etc/freeradius/users file, add a client in the /etc/freeradius/clients.conf and start the service up (I recommend debug). The formatting for adding a user/client is actually documented well. Remember, a user is a user on your router/device, a client is the configuration knob required on your RADIUS gateway (in this case, also a router). You need a secret phrase between the router and RADIUS server to allow them to talk, as well as a user defined on both sides (the subject of the talk between client and server).

Step-by-Step guide to installing Smokeping on Ubuntu 14.04 LTS

Installing Smokeping on Ubuntu used to be a total breeze. Since 14.04 however, it’s been a bit of a mission.

This guide assumes a fresh out of the box Ubuntu install. I’m using the 64bit Server variety, but this should work on any 14.04 system.

UPDATE – It also works perfectly on Ubuntu 16.04 LTS Server :~)

Note – I am a noob with Apache and a relative noob with Linux, but even I got it to work fairly painlessly.

Step 1: install smokeping (aaand you’re done).
 sudo apt-get install smokeping -y

Step 2:  normally, you’d be done by now.. But things have changed.
 sudo nano /etc/smokeping/config.d/pathnames

You’re going to want to go into the pathnames file and comment out the line about mail. Setting up mail to work with smokeping is outside the scope of this post. Because I’m lazy.

Do this: #sendmail = /usr/sbin/sendmail
Then hit CTRL-O, Enter, CTRL-X. That’s how you save a file in Nano, I won’t stick that bit in again.

Step 3: If you made it this far, you’re going to be fine. Start ‘er up…
sudo service smokeping start

Step 4: Head over to a web browser and enter the IP of your server/cgi-bin/smokeping.cgi
(Hint: it will fail, giving you a 404 error).
(Hint 2: you can find the IP address of your server by entering ifconfig, it’s typically eth0)

Step 5: It’s not working because you’re missing a couple of things. One is a slight config change in a smokeping config file,  one is some missing symlinks.. the other is (likely) the cgi module for Apache2 isn’t enabled. Fix it!

sudo nano /etc/smokeping/config.d/General

Edit the line that has cgiurl in it to read like:
cgiurl = http://10.10.10.10/cgi-bin/smokeping.cgi

Save and quit nano.

Next you want to edit the following Apache config file:
sudo nano /etc/apache2/conf-available/serve-cgi-bin.conf

Under the lines

“Require all granted
     </Directory>”

You want to add:

ScriptAlias /smokeping/smokeping.cgi /usr/lib/cgi-bin/smokeping.cgi
 Alias /smokeping /usr/share/smokeping/www
 <Directory “/usr/share/smokeping/www”>
 Options FollowSymLinks
 </Directory>

(Save and quit editing the file)

Finally, enable CGI:
sudo a2enmod cgi

Step 6: Kind of threw 3 things into step 5 there, whoops. Anyway now you want to restart smokeping and apache, just to see if you broke anything:
sudo service apache2 restart
sudo service smokeping restart

Now head back to your browser, throw in 10.10.10.10/cgi-bin/smokeping.cgi

It should work… If not, drop me a line.

 

3DS Streetpass on Mikrotik RouterOS

Quick and hacky – that’s the way you have to Streetpass in New Zealand, as you get so few legit matches out and about.

To setup your own home Streetpass service is easy.

  • Setup a new SSID on your router, as a virtual ap – call it attwifi
  • Dupe the MAC address on the new virtual AP to 4E:53:50:4F:4F:46
  • Stick that SSID in your local bridge and setup some security
  • Join your 3DS to the attwifi SSID and away you go

Simple as that!

How to reach PPPoE bridge from Mikrotik

When using a Mikrotik router (or any other decent home router) as your PPPoE client, it’s good to be able to keep access to the ADSL/VDSL modem in-line to allow diagnostics, additional configuration etc. To avoid a situation where the router is essentially double-NATing all the packets going across the WAN link, the ideal setup is a secondary IP address on the router’s WAN interface, that is handled separately.

Below is a crude drawing of my home setup. The Draytek Vigor 130 is acting as a VDSL modem, bridging the PPPoE connection across to the Mikrotik RB2011UAS-2HnD-IN which is ‘dialing’ the PPPoE connection back to my ISP. The ISP dishes out an IP address which lands directly on the router, passing through the Draytek. On the LAN side of the router, is a pretty boring DHCP subnet (with DNS setup as mentioned here).

To setup the router/modem to allow access to both (without unplugging the router to get back to the modem) – you can do the following:

First, add an IP address to the modem:

(On Draytek, using GUI, added 192.168.2.1 255.255.255.252)

Now, add the corresponding interface on the Mikrotik (access via SSH, note below is only 2 lines of config):

/ip address
add address=192.168.2.2/30 comment="To get to VDSL modem" interface=ether1-gateway network=192.168.2.0

This will add the other end of the /30 network to the ether1-gateway (physical) interface on the Mikrotik router.

Now, all we need to do is tell the router that it’s OK to NAT on that address, on that interface:

/ip firewall nat
add action=masquerade chain=srcnat out-interface=ether1-gateway

If all went to plan, you should be able to ping 192.168.2.1 from your Mikrotik:

[admin@burnett-home] > ping 192.168.2.1
HOST                                     SIZE TTL TIME  STATUS
192.168.2.1                                56 255 0ms
192.168.2.1                                56 255 0ms
    sent=2 received=2 packet-loss=0% min-rtt=0ms avg-rtt=0ms max-rtt=0ms

All done, now you can browse to both the router IP for GUI config, and get to the VDSL modem’s config page as well.

Help on this one came from the DD-WRT wiki.

Mikrotik Firewall – For Home Users

Mikrotik make routers that are affordable enough for the home user market, but are quite powerful and come without too many training wheels. I’m using the RB2011UAS-2HnD-IN as a home router, wireless AP and firewall. It’s powerful and configurable enough to do pretty much anything I’ve thrown at it – but out of the box it’s probably a bit too open to attacks from randos.

Here’s a firewall script I’ve deployed on the home gateway, with the following parameters

Home LAN Subnet – 192.168.88.0/24
Home LAN Gateway – 192.168.88.1

/ip firewall filter
add chain=input comment="default configuration" protocol=icmp
add chain=input comment="default configuration" connection-state=established
add chain=input comment="default configuration" connection-state=related
add action=drop chain=input comment="default configuration" in-interface=sfp1-gateway
add action=drop chain=input comment="default configuration" in-interface=ether1-gateway
add chain=forward comment="default configuration" connection-state=established
add chain=forward comment="default configuration" connection-state=related
add action=drop chain=forward comment="default configuration" connection-state=invalid
add action=drop chain=input comment="Drop Invalid connections" connection-state=invalid
add chain=input comment="Allow Established connections" connection-state=established
add chain=input comment="Allow ICMP" protocol=icmp
add chain=input in-interface=!ether1-gateway src-address=192.168.88.0/24
add action=drop chain=input comment="Drop everything else"
add action=drop chain=forward comment="drop invalid connections" connection-state=invalid protocol=tcp
add chain=forward comment="allow already established connections" connection-state=established
add chain=forward comment="allow related connections" connection-state=related
add action=drop chain=forward comment="block bogon ip addresses" src-address=0.0.0.0/8
add action=drop chain=forward comment="block bogon ip addresses" dst-address=0.0.0.0/8
add action=drop chain=forward comment="block bogon ip addresses" src-address=127.0.0.0/8
add action=drop chain=forward comment="block bogon ip addresses" dst-address=127.0.0.0/8
add action=drop chain=forward comment="block bogon ip addresses" src-address=224.0.0.0/3
add action=drop chain=forward comment="block bogon ip addresses" dst-address=224.0.0.0/3
add action=jump chain=forward comment="make jumps to new chains" jump-target=tcp protocol=tcp
add action=jump chain=forward comment="make jumps to new chains" jump-target=udp protocol=udp
add action=jump chain=forward comment="make jumps to new chains" jump-target=icmp protocol=icmp
add action=drop chain=tcp comment="deny TFTP" dst-port=69 protocol=tcp
add action=drop chain=tcp comment="deny RPC portmapper" dst-port=111 protocol=tcp
add action=drop chain=tcp comment="deny RPC portmapper" dst-port=135 protocol=tcp
add action=drop chain=tcp comment="deny NBT" dst-port=137-139 protocol=tcp
add action=drop chain=tcp comment="deny cifs" dst-port=445 protocol=tcp
add action=drop chain=tcp comment="deny NFS" dst-port=2049 protocol=tcp
add action=drop chain=tcp comment="deny NetBus" dst-port=12345-12346 protocol=tcp
add action=drop chain=tcp comment="deny NetBus" dst-port=20034 protocol=tcp
add action=drop chain=tcp comment="deny BackOriffice" dst-port=3133 protocol=tcp
add action=drop chain=udp comment="deny TFTP" dst-port=69 protocol=udp
add action=drop chain=udp comment="deny PRC portmapper" dst-port=111 protocol=udp
add action=drop chain=udp comment="deny PRC portmapper" dst-port=135 protocol=udp
add action=drop chain=udp comment="deny NBT" dst-port=137-139 protocol=udp
add action=drop chain=udp comment="deny NFS" dst-port=2049 protocol=udp
add action=drop chain=udp comment="deny BackOriffice" dst-port=3133 protocol=udp
add chain=icmp comment="echo reply" icmp-options=0:0 protocol=icmp
add chain=icmp comment="net unreachable" icmp-options=3:0 protocol=icmp
add chain=icmp comment="host unreachable" icmp-options=3:1 protocol=icmp
add chain=icmp comment="host unreachable fragmentation required" icmp-options=3:4 protocol=icmp
add chain=icmp comment="allow source quench" icmp-options=4:0 protocol=icmp
add chain=icmp comment="allow echo request" icmp-options=8:0 protocol=icmp
add chain=icmp comment="allow time exceed" icmp-options=11:0 protocol=icmp
add chain=icmp comment="allow parameter bad" icmp-options=12:0 protocol=icmp
add action=drop chain=icmp comment="deny all other types"
/ip firewall nat
add action=masquerade chain=srcnat comment="default configuration" out-interface=pppoe-out1

This firewall script can be used to somewhat lockdown access to your router/home LAN without too much struggle. It’s probably missing a few bits and pieces, but seems to have put and end to hackers trying to brute-force root access via SSH over the internet – so far.

Unblock-US and Mikrotik RB2011UAS-2HnD-IN

If you live somewhere out of the way, like New Zealand, sometimes it pays to use a DNS-proxying service like Unblock-US. Reasons for this are circumventing some draconian geo-blocking rules, but I won’t go into that here.

Anyway – if you want to setup your Mikrotik router to use DNS addresses other than your ISPs provided ones (and make use of the DNS cache offered by the router) – follow these steps:

First, SSH into your router (something like admin@routerIP). I use Putty to do this in Windows. Then enter the following lines:

/ip dns
set allow-remote-requests=yes servers=111.118.175.56,118.127.33.48,208.122.23.23
/ip dns static
add address=192.168.88.1 name=router

That adds the 3 Unblock-US DNS addresses they specify to be the router’s DNS servers. You can check it worked by the following command:

/ip dns print

Now, the DNS is set, but nothing on your home LAN is going to be interested until you either statically point each device to use the DNS address of your router (192.168.88.1 by default), or tell the router’s DHCP server where to get its DNS info from. The latter is the most elegant option. Do it by:

/ip dhcp-server
add address-pool=dhcp disabled=no interface=bridge-local lease-time=10m name=default
/ip dhcp-server network
add address=192.168.88.0/24 comment="default configuration" dns-server=192.168.88.1 gateway=192.168.88.1 \
    netmask=24

Here, we go into the DHCP server config and set up a DHCP pool on the local bridged interfaces (the locally connected devices, including wifi hosts). Then, we tell the DHCP server to use the router’s IP as its DNS source. Quite often this is already set by default.

Now, you can browse around a bit, using the Unblock-US DNS servers to get to where you want to go. You can check on the DNS Cache by entering:

/ip dns cache print

It should have a few hundred entries after a minute or so of cruising around the internet.

The final (and crucial) step, is to disable ‘Peer DNS’ on your WAN interface. I did this by disabling the option on my pppoe-out1 interface

/interface pppoe-client 
use-peer-dns=no

Some info from here.