Upgrading Junos issue – not enough space

Quick note, mostly for my own reference down the track.

I have in the lab a MPC3E-NG FQ and a MIC3-100G-DWDM card. To those of you not Juniper ensconced, that’s a chassis-wide slot and a 100Gbit/s DWDM (tunable optic) card. Wowee, I hear you say. Anyway, the 100G DWDM card requires a fairly spicy new version of Junos, one with a new underlying FreeBSD core at the heart. My lab was running on Junos 14.1R5.5, an already pretty recent version – but for 100G across my lab I need to use the DWDM card, inconjunction with some Infinera DWDM kit for good measure.

Normally, a Junos upgrade is fairly painless. In this case however, I was getting errors that I couldn’t understand. Here is how I pushed past them.

When going from 14.x to 15.1F6, Juniper don’t recommend running validation of the software release against your config.

james@mxZ.lab.re1> request system software add /var/tmp/junos-install-mx-x86-64-15.1F6.9.tgz no-validate re1

When I ran this on one RE (RE0) it went through just fine. On RE1 however, I got this:

Installing package ‘/var/tmp/junos-install-mx-x86-64-15.1F6.9.tgz’ …
Verified manifest signed by PackageProductionEc_2016
Verified manifest signed by PackageProductionRSA_2016
Verified manifest signed by PackageProductionRSA_2016
Verified contents.iso
Verified issu-indb.tgz
Verified junos-x86-64.tgz
Verified kernel
Verified metatags
Verified package.xml
Verified pkgtools.tgz
camcontrol: not found
camcontrol: not found
camcontrol: not found
Verified manifest signed by PackageProductionEc_2016
Saving the config files …
NOTICE: uncommitted changes have been saved in /var/db/config/juniper.conf.pre-install
tar: contents/jpfe-wrlinux.tgz: Wrote only 0 of 10240 bytes
tar: Skipping to next header
tar: Archive contains obsolescent base-64 headers
tar: contents/jplatform-ex92xx.tgz: Wrote only 0 of 10240 bytes


tar: Skipping to next header
tar: Error exit delayed from previous errors
ERROR: junos-install fails post-install

This looks a bit like something has gone wrong, but it’s not immediately obvious what it is. I am using a .tgz of Junos 15 that is sitting on my RE1’s /var partition, and it has 2.1 GB hard-drive space free (show system storage)… Queue a few hours of headscratching.

Turns out, my assumption on how much space is actually required when upgrading from Junos 14 to 15 was completely under. I started with about 2GB free, but when I was finally successful I had 11GB free. Once the install was complete, I was down to 7.7G, which means the installation process uses up 3.3GB all by itself. I guess that’s not crazy, as the tarball is 1.9GB to begin with, but the error output didn’t make it clear enough for me, a stupid man.

Here is how I overcame the obvious and got Junos 15.1F6 installed 🙂

1: Jump into shell

james@mxZ.lab.re1> start shell

2: Become root

james@mxZ.lab.re1> su
<enter password>

3: Find all the files that are > 500MB. We’re looking for things in /var

root@mxZ% find / -size +500000

4: Delete any of the old .tgz files from previous releases. If you want to keep them, SCP them off first.

root@mxZ% rm /var/sw/pkg/jinstall64-12.1R2.9-domestic-signed.tgz

etc etc

5: Check you now have > 3.5 GB free. Or go all out like me and have 11GB free, whatever.

6: Upgrade Junos and get back to work!

WordPress on a VPS.. Ugh.

As part of my dumb journey to self-host things (well, on a VPS I pay for), I fired up Apache and chucked a virtual host on it. The plan will be to move all my hosted sites to this server, but for now I’m starting with a fresh WordPress install for a family member’s B&B website.

In the mix we have:

  • Apache2
  • Ubuntu 14.04 LTS 64bit
  • php5
  • MySQL
  • phpmyadmin
  • WordPress latest
  • Sweat coming out of my face

So the basic LAMP install was done without a hitch. Where WordPress started to suck was permissions. By default, somehow, I installed WordPress with the permissions set really way wide open (777 on directories).. I used some commands in the WordPress codex (inside my /var/www/sitename/public_html folder)..

james@chappie:~$ sudo find . -type f -exec chmod 664 {} +
james@chappie:~$ sudo find . -type d -exec chmod 775 {} +
james@chappie:~$ chmod 400 wp-config.php

So, there’s that. Then I found a user couldn’t upload a theme or plugin, because for some reason (despite www-data, the Ubuntu Apache user, having group rights to the files..) WordPress couldn’t write to wp-content/, the folder these things go in by default.

If I ran ‘su www-data touch check’ inside wp-content, the file “check” was created, so it wasn’t a permissions issue as far as I could tell. Weird. I ended up fixing it by explicitly telling WordPress to allow ‘direct’ uploading by adding:

define(‘FS_METHOD’, ‘direct’);

To the wp-config.php file.. All of a sudden.. It works.

I had read elsewhere (a 3 year old mysteriously written comment on WordPress’ own community site) that changing PHP’s config to use FastCGI/php-fpm was the solution.. I am still (and probably always will be) a noob with these web technologies, so I wasn’t really 100% sure what I was doing. I also failed, lucky ‘FS_METHOD’ being set to direct was what I wanted.


I found that changing the ‘server api’ of PHP5 to FPM/FastCGI (and removing the aformentioned FS_METHOD) also works. I followed the steps listed here (JDawg’s answer): http://askubuntu.com/a/527227/521633

Running your own VPS kind of sucks

I pay a good chunk of change to Dreamhost every year, I have done since I was old enough to have a credit card. It’s handy. They are a pretty chill hosting provider, by and large. I host about 8 or 9 WordPress sites. Some are mine, some for friends. As I get better at unix’y stuff, I’m becoming more of a cheapskate and am thinking that $140NZ or whatever Dreamhost costs per year is a bit steep for a shared hosting platform. I have trialled their more dedicated VPS offering and while it kills performance-wise, it’s too expensive for someone hosting sites on behalf of others for free…

A while back I got a good deal on Zappie Host, based in New Zealand, where most of my ‘clients’ are also based. It’s a dedicated VPS, 2 CPU cores and a gig of RAM for about $7NZ a month. I am also on a 50% off deal for the first year, so it’s a good deal. I’m paying for it in parallel with DH so I can get my shit together and migrate. Part of the motivation for sorting it all out before my next DH payment is due is financial, as I’m paying for 2 hosting solutions and not getting much benefit from the combination.

Some challenges:

  • Picking a web server (Apache or NGINX?!)
  • Picking a database (MariaDB or MySQL!?)
  • To use a cPanel or not?
  • Security
  • Mail (jeesus christ, email is complex!)
  • Virtualisation/separation of different client sites
  • Backups!?
  • Remote access for users
  • Unknown unknowns
  • DNS
  • Goddamn email
  • Resource monitoring

I think I need a coffee. What I want to do is use this list and knock out some posts detailing the crap you need to put up with to save a few bucks and call yourself a server administrator 🙂

Time for you and time for me,
And time yet for a hundred indecisions,
And for a hundred visions and revisions,
Before the taking of a toast and tea.

– T.S. Eliot (lol)

Firing up NetBox

Over at Packet Life, stretch has been talking about his/Digital Ocean’s cool new IPAM/DCIM. It’s open source, being developed like crazy and has some interesting features (for me, the ability to document my lab setup which is getting out of hand seems like a good place to start).

I ran through the installer on Github and squirted the install onto a fresh Ubuntu 14.04 LTS server on AWS. Here are my thoughts…

  • The install instructions were very well written for a fresh OSS product, pretty much nothing failed the first time through. This is reminiscent of other Digital Ocean documentation I have read. I did need to jump out of editing a config file to generate a key, but that’s minor.
  • From an absolute beginner with Linux pov (not quite the boat I’m in, but a boat I was fairly recently in) – there are a few things missing (like when to sudo/be root or not per command). It’s not a noobs setup guide, but it is pretty easy to follow otherwise.
  • The installation/setup takes around 10-15 mins including generating the AWS EC2 host
  • The interface of NetBox looks lovely and clean

That was pretty easy and quite smooth to install. Now it’s up and running I can’t wait to document my lab and post the results up here.