Archive for Linux

Let’s Encrypt – Doing it Properly

A while ago I made a passing comment about replacing StartSSL using Let’s Encrypt and while it’s true you can use it in the way I described, it’s not as simple as the command would imply and there are a number of pitfalls – not least that if you side-step most of them by going a truly manual route, your efforts only last for 3 months – let’s encrypt certificates expire after only 3 months.

What this all boils down to is that in order to make proper use of this excellent free service you have to automate the process of obtaining and renewing certificates. Not just that though, because their python tooling (certbot) will do this in a general fashion for you, but to truly implement a hands off one size fits all solution. I articulated this as the following requirements.

  • Tool must check for impending expiry and renew in time to avoid nag e-mails
  • Support should be available for adding new certificate requests to the “pool”, not just to renew existing ones
  • A fixed, permanent web server must not be required
  • Servers running on non-standard ports must be supproted
  • Servers not ordinarily accessible to the public must be supported
  • Firewall must not be required to always be open to Let’s Encrypt servers
  • Tool must support servers who’s control IP (SSH etc.) differs from that to which an SSL certificate is to be assigned (A record of SSL domain)

A evening of hacking around on a Raspberry Pi to get certbot running and some hasty coding in bash revealed a solution which essentially, does the following, in order, for each domain requiring a certificate.

  1. Checks to see if the domain has a cert and if it’s expiring
  2. Creates a remote directory from which files will be served as part of the renewal process
  3. Inserts a temporary firewall rule to allow Lets’ Encrypt servers to validate the served files
  4. Starts a temporary lightweight webserver to serve said files and runs it on a port that Let’s Encrypt supports (80)
  5. Mounts this served directory locally on the certbot machine so that certbot can load files into it
  6. Runs certbot to create/renew the certificate
  7. Unmounts the temporary server directory
  8. Stops the temporary webserver
  9. Removes the temporary firewall rule
  10. Cleans up temporary files
  11. Load the certificate files onto the remote server over SSH
  12. Bounces any services that use them so that they are picked up (e.g. Apache)

Before I share the script, a few words of warning and a few pointers

  • Everything needs to run as root
  • I mounted a NAS share of my certificate files so that they were stored and backed up centrally. Certbot expects to find everything in /etc/letsencrypt so that will need to be your mount point if you use the NAS option
  • The script assumes that the server on which it is running has SSH keys setup on all hosts it needs to connect to such that passwordless root is possible.

If you don’t know how to do any of these things then you probably shouldn’t try using the script. It’s not that you won’t be capable or anything, rather that there are a lot of moving parts and basic Linux knowledge that would allow you to achieve these 3 points will go a long way in helping you to debug any issues with the script.

Finally then, the script; configured for two dummy domains, running on the same box, one serving from apache, the other a custom process.

# Configuration section
remotePort=80 # Lets encrypt only supports 80 - ouch - we'll need to stop servers while doing this
domains=('' '')
hosts=('' '')
sshHosts=('' '')
sslPathOnServer=('/etc/apache2/ssl/' '/etc/specialprocess/ssl/')
keyFileNameOnServer=('www.key' 'pub.key')
certFileNameOnServer=('www.crt' 'cert.crt')
chainFileNameOnServer=('www-chain.pem' 'chain.pem')
serverStopCommand=('/etc/init.d/apache2 stop' '/etc/init.d/specialprocess stop')
serverStartCommand=('/etc/init.d/apache2 start' '/etc/init.d/specialprocess start')

# Work is done here, no further edits!
for domain in ${domains[@]}; do
        # Ascertain if the certificate will expire in 3 weeks or less - stops lets encrypt nagging e-mails
        if [ -d "$certPath$domain" ]
                openssl x509 -in $certPath$domain/cert.pem -checkend $(( 86400 * 21 )) -noout
        if [[ $result == 1 ]]
                # Report the status
                if [[ $new == 1 ]]
                        echo "Certificate for $domain [New]"
                        echo "Running certificate creation process..."
                        echo "Ceritificate for $domain [Expiring]"
                        echo "Running renewal process..."
                # Start renewal process; create remote directory, iptables rule, start remote server
                echo -n "       Setting up iptables rules on remote server, stopping server process using certs & starting temporary micro server..."
                rule="INPUT 1 -p tcp -m tcp --dport $remotePort -d ${hosts[i]} -j ACCEPT"
                ssh root@${sshHosts[i]} << ENDSSH > /dev/null 2>&1
                eval ${serverStopCommand[i]}
                mkdir /tmp/$domain
                iptables -I $rule
                cd /tmp/$domain
                python -m SimpleHTTPServer $remotePort > /dev/null 2>&1 &
                echo " Done"

                # Mount remote directory locally over SSHFS
                echo -n "       Creating mount to directory served by temporary micro server..."
                if [ -d /tmp/certbot ]
                        rm -rf /tmp/certbot
                mkdir /tmp/certbot
                sshfs ${sshHosts[i]}:/tmp/$domain /tmp/certbot
                echo " Done"

                # Call letsencrypt to renew/create
                echo -n "       Call letsencrypt to renew the certificate in our store..."
                letsencrypt certonly --webroot -n -w /tmp/certbot -m $email -d $domain > /dev/null 2>&1
                echo " Done"

                # Unmount remote SSHFS directory
                echo -n "       Unmounting micro server directory and cleaning mount point..."
                umount /tmp/certbot
                rm -rf /tmp/certbot
                echo " Done"

                # SCP certificate/chain/key files to remote host
                echo -n "       Copying new certificate files to the server..."
                scp $certPath$domain/cert.pem ${sshHosts[i]}:${sslPathOnServer[i]}${certFileNameOnServer[i]} > /dev/null 2>&1
                scp $certPath$domain/privkey.pem ${sshHosts[i]}:${sslPathOnServer[i]}${keyFileNameOnServer[i]} > /dev/null 2>&1
                scp $certPath$domain/chain.pem ${sshHosts[i]}:${sslPathOnServer[i]}${chainFileNameOnServer[i]} > /dev/null 2>&1
                echo " Done"

                # Cleanup remote host; stop temp server, reload default iptables, cleanup temp directory, restart server process
                echo -n "       Stop micro server, clean up iptables rule & start server process using certs... "
                ssh ${sshHosts[i]} << ENDSSH > /dev/null 2>&1
                kill \`ps -ef | grep SimpleHTTPServer | grep -v grep | awk '{ print \$2; }'\`
                iptables -D INPUT 1
                rm -rf /tmp/$domain
                eval ${serverStartCommand[i]}
                echo " Done"

                # Confirm completion
                if [[ $new == 1 ]]
                        echo "Certificate for $domain [Created]"
                        echo "Creation process complete"
                        echo "Renewal process complete"
                        echo "Certificate for $domain [Renewed]"
                # Certificate needs no renewal, indicate as such
                echo "Certificate for $domain [OK]"

You’ll need to modify the variables/arrays at the start to get it working, some are obvious, but I’ll explain them anyway.

  • email : This is used to create, recover and access your account. Once created, the private key for accessing your account will be stored in the let’s encrypt home directory (usully /etc/letsencrypt)
  • certPath : The path to the /live directory in your let’s encrypt home directory, usually /etc/letsencrypt/live
  • remotePort : Leave this at 80; it’s just here to help setup iptables rules – it cannot be changed in let’s encrypt config
  • domains : An array of domains for which you want certificates
  • hosts : An array of IP addresses corresponding to the aforementioned certificates – keep the ordering the same!
  • sshHosts : An array of IP addresses used for administering each of the aforementioned domains; for example you may have several web servers/domains all being served from the same physical host – this would then be an array of identical IPs, corresponding in array size with the number of domains
  • sslPathOnServer : What is the file system path to each domain’s certificates on the host server
  • keyFileNameOnServer : Within that directory, what is the key file name
  • certFileNameOnServer : Within that directory, what is the certificate file name
  • chainFileNameOnServer : Within that directory, what is the chain file name
  • serverStopCommand : For each domain, what command stops the server process the domain is used for
  • serverStartCommand : Likewise, what is the command to start the server process again

That’s it! Best of luck if you choose to go this way and do let me know if the script is useful to you.


Raspberry Pi MAC Address Changing on Reboot

I recently did a apt-get dist-upgrade on my Raspberry Pi running Raspbian 8 and when I rebooted the device appeared to drop clean off my network. I was away from home at the time so I decided to fix it when I was next on the premises – it wasn’t a hugely important device to have running 24/7.

When I looked at the device in person I realised that it was in fact connected to the network, but with a different IP address to that it had held before. Having MAC address based IP allocation, this seemed rather strange. Looking at the MAC address in ifconfig revealed the problem – it had changed! Thinking this to just be a fluke, I rebooted. Now, not only was the MAC still different to that which it should have been, it had changed a third time!

Investigating what had happened during the upgrade revealed that both the bootloader and the kernel had both been upgraded, but for whatever reason, the new kernel wasn’t being loaded. Modifying the /boot/config.txt as follows along with a reboot fixed this discrepancy and lo and behold the MAC returned to it’s former value and the device picked up the IP address it was supposed to have over DHCP


initramfs initrd.img-4.4.0-1-rpi followkernel
# to enable DeviceTree, comment out the next line


initramfs initrd.img-4.9.0-2-rpi followkernel
# to enable DeviceTree, comment out the next line

It’s worth pointing out that you may have a similar problem with a similar cause but slightly different kernel versions to me. If this is the case, or to check if it is, verify the old/new versions in /boot as follows

ls -lha /boot | grep initrd.*rpi

Which will reveal your old/new versions

-rwxr-xr-x 1 root root 5.7M Jul 3 18:26 initrd.img-4.4.0-1-rpi
-rwxr-xr-x 1 root root 13M Jul 3 18:46 initrd.img-4.9.0-2-rpi

You should then modify /boot/config.txt from the old version to the new version as per my above example.


Failed to connect to lockdownd

After a recent holiday I wanted to download the photos from my iPhone – something I’ve long been able to do in Linux. Upon following my well trodden process, I was greeted with an unfamiliar error message

$ ifuse /media/kieran/iphone
Failed to connect to lockdownd service on the device.
Try again. If it still fails try rebooting your device.

Some reading around the subject revealed that encryption needed to be disabled on the libimobiledevice module that is used under Linux to pair with the iPhone. Rather than go to the effort of modifying the source myself, some helpful chap has already done this and published the code and usage guide to GIT. Hope this pointer helps someone who is stuck trying to access their phone.


PPTP VPN in Ubuntu 15.10

Recently I had cause to connect to my home VPN from my laptop, being out of town and on a connection not part of my usual trusted network. I tried to connect as usual using PPTP via the network manager on the Xubuntu desktop but no dice – the connection was failing every time. After establishing my iPhone was able to connect from the same network, I realised that the last time my laptop had successfully connected to the VPN was prior to my upgrade to Ubuntu 15.10 and therefore something must have changed with the default settings under the hood to cause the problem.

To cut a long story short, after much digging and hair pulling, I established that you need to enable 3 kernel modules to get the PPTP VPN client back up and running in the latest version of Ubuntu. Add the following to /etc/modules and reboot


After that you should find that your PPTP VPN connection invoked in the usual way from the network manager no longer fails. Happy days!

Comments off    

Export Public Key From pkcs12 Container

Everytime I renew my S/MIME certificate I always seem to wind having to consult the OpenSSL man pages to ascertain the approrpiate command to use in order to extract the certificate from the PKCS12 file I get from my certificate authority so I can share said certificate on my contact page. As I’m sure I’m not the only one who winds up having to look this up, I’m sharing it here.

openssl pkcs12 -in container.p12 -clcerts -nokeys -out public.pem

If you then investigate the contents of the PEM file inside a text editor you’ll find your certificate between the BEGIN and END lines – you should share this content on your website or directly with contacts so that your signed e-mails can be verified as to their origin and also so that contacts can encrypt mail such that it can only be read by you if required.

Comments off    

Handling the Heartbleed Bug

So when news of the Heartbleed Bug hit the tech news sites and security bulletins I decided to read up on it when I got home from work and go from there. While it’s true that some news sites aimed at the general public have gone rather over the top about this and in many cases got their facts and recommendations plain wrong, my evening reading was not that enjoyable – this is indeed a very serious bug and is likely to affect everyone in some way shape or form. Note that I said affect, and not compromise – this is important – there was and still is no evidence that the bug was being exploited in the wild.

For those with a technical inclination, the register has done a nice little round up of how the bug works. The non technical among you may find this heartbleed cartoon illustration over on XKCD more useful.

Anyway, I thought I’d share with you the steps I have taken and will be taking going forward to help mitigate the effects of the bug. I’ve laid out the steps in order, note that I’ve focused on external services first – this was deliberate – no sense in using external services to help me fix my own infrastructure if they were not patched yet.

External Infrastructure

  • Verified with my hosting company that their servers were not affected. Xilo confirmed to all customers that their OpenSSL version was prior to the appearance of the bug. This ensured all keys stored in their infrastructure and used by me are secure
  • Ensured that my cloud based password management system was unaffected. LastPass were very open and detailed with their customers in explaining that despite their servers having been affected and subsequently patched and their certificates re-issued, a combination of passwords never being sent to their systems without an additional layer of encryption to which they never see the key and Perfect Forward Secrecy, customer data was and remains secure.
  • Verified that my certificate provider’s online portal is currently patched and protected. StartSSL have confirmed that their infrastructure was never vulnerable to the attack.

Own Infrastructure

  • Patched all servers I administer that were vulnerable – check yours here

    apt-get update
    apt-get upgrade

  • Re-generated all SSH host keys on vulnerable hosts (remember to leave the pass phrase blank)

    ssh-keygen -t dsa -f /etc/ssh/ssh_host_dsa_key
    ssh-keygen -t rsa -f /etc/ssh/ssh_host_rsa_key

  • Revoked certificates that were in use on previously vulnerable hosts. Some providers may charge for this but you’re not fully secure without it!
  • Re-issued certificates that had been revoked using new keys. Generate these and CSR as follows

    openssl genrsa -out server.key 4096
    openssl req -new -key server.key -out server.csr

  • Restarted all services using OpenSSL. In many cases this will be done by the package manager during the upgrade process but if you’ve changed keys etc. a restart will be needed to pick this up

    /etc/init.d/ssh restart
    /etc/init.d/apache2 restart


  • Given that changing a password prior to a site admin both patching the server and revoking/re-issuing their certificate is a pointless exercise, I plan on relying on a combination of the LastPass checker and conversations with web masters directly about if I should change passwords or not. I’m expecting a large number to need changing but hopefully it can be done at a relaxed pace over the next few weeks as and when news becomes available regarding the vulnerability or otherwise of a site.


  • Confirmed, via the LastPass checker, that PayPal have never been vulnerable due to the software they use for SSL
  • While it’s possible that credit card numbers, bank details and indeed other personal details may have been leaked via the bug, I deemed that replacing bank cards etc. was overkill. After all most banks offer fraud cover and my subscription to credit report monitoring includes tools to help manage any increased risk of identity theft. In any case, many personal details sent in this way are not easily changed, in fact only a credit card number can really be replaced – bank details, national insurance number, address etc. are all set in stone.

Hopefully this overview is helpful to those who are confused!

Comments (2)    

SecurityException in Application

When carrying out a recent upgrade of suPHP I encountered the following error in my apache error logs and PHP pages were not being served.

SecurityException in Application.cpp:511: Unknown Interpreter: php

Turns out the fix is a simple one; the syntax of the config file has change between versions and it’s simple a case of replacing the following lines:

;Handler for php-scripts

;Handler for CGI-scripts

With the following (note the addition of double quotes)

;Handler for php-scripts

;Handler for CGI-scripts

A quick bounce of apache and you should be back and kicking

Comments off    

Removing kernel from OpenVZ Container

When I recently upgraded an OpenVZ container in-situ over SSH, I had a few issues with the upgrade path. One of these was that the upgrade process called for the installation of a new kernel, however a container doesn’t need one and in my case couldn’t handle installing one either and thus caused the upgrade process to error out.

The fix was to force remove the kernel, both from the dpkg directory structure (in this case I copied the files to /tmp/ just in case) and using dpkg it’s self. The following commands as root:

mv /var/lib/dpkg/info/linux-image-3.2.0-58-generic.* /tmp/
dpkg --remove --force-remove-reinstreq linux-image-3.2.0-58-generic

Resuming the dist-upgrade after this process was a success

Comments off    

Kworker High CPU

After a recent kernel upgrade on my home server I noticed that Kworker was hogging most of the CPU after boot and wasn’t dying down over time. Research online lead me to believe this was due to an ACPI interrupt storm creating a high load. I tested a fix for this by adding the following line to root crontab

@reboot echo "disable" > /sys/firmware/acpi/interrupts/gpe00

This essentially serves to switch off ACPI and upon reboot I found that CPU usage had indeed returned to normal. As setting ACPI to off has no impact on my home server and what it is used for I decided to leave this tweak in place and to check to see if it was still required when future kernel upgrades come around – this has been a widely reported bug on Ubuntu launchpad.

Comments off