A while ago I made a passing comment about replacing StartSSL using Let’s Encrypt and while it’s true you can use it in the way I described, it’s not as simple as the command would imply and there are a number of pitfalls – not least that if you side-step most of them by going a truly manual route, your efforts only last for 3 months – let’s encrypt certificates expire after only 3 months.

What this all boils down to is that in order to make proper use of this excellent free service you have to automate the process of obtaining and renewing certificates. Not just that though, because their python tooling (certbot) will do this in a general fashion for you, but to truly implement a hands off one size fits all solution. I articulated this as the following requirements.

• Tool must check for impending expiry and renew in time to avoid nag e-mails
• Support should be available for adding new certificate requests to the “pool”, not just to renew existing ones
• A fixed, permanent web server must not be required
• Servers running on non-standard ports must be supproted
• Servers not ordinarily accessible to the public must be supported
• Firewall must not be required to always be open to Let’s Encrypt servers
• Tool must support servers who’s control IP (SSH etc.) differs from that to which an SSL certificate is to be assigned (A record of SSL domain)

A evening of hacking around on a Raspberry Pi to get certbot running and some hasty coding in bash revealed a solution which essentially, does the following, in order, for each domain requiring a certificate.

1. Checks to see if the domain has a cert and if it’s expiring
2. Creates a remote directory from which files will be served as part of the renewal process
3. Inserts a temporary firewall rule to allow Lets’ Encrypt servers to validate the served files
4. Starts a temporary lightweight webserver to serve said files and runs it on a port that Let’s Encrypt supports (80)
5. Mounts this served directory locally on the certbot machine so that certbot can load files into it
6. Runs certbot to create/renew the certificate
7. Unmounts the temporary server directory
8. Stops the temporary webserver
9. Removes the temporary firewall rule
10. Cleans up temporary files
11. Load the certificate files onto the remote server over SSH
12. Bounces any services that use them so that they are picked up (e.g. Apache)

Before I share the script, a few words of warning and a few pointers

• Everything needs to run as root
• I mounted a NAS share of my certificate files so that they were stored and backed up centrally. Certbot expects to find everything in /etc/letsencrypt so that will need to be your mount point if you use the NAS option
• The script assumes that the server on which it is running has SSH keys setup on all hosts it needs to connect to such that passwordless root is possible.

If you don’t know how to do any of these things then you probably shouldn’t try using the script. It’s not that you won’t be capable or anything, rather that there are a lot of moving parts and basic Linux knowledge that would allow you to achieve these 3 points will go a long way in helping you to debug any issues with the script.

Finally then, the script; configured for two dummy domains, running on the same box, one serving from apache, the other a custom process.

#!/bin/bash
# Configuration section
email=your.email@address.com
certPath=/etc/letsencrypt/live/
remotePort=80 # Lets encrypt only supports 80 - ouch - we'll need to stop servers while doing this
domains=('one.domain.com' 'two.domain.com')
hosts=('1.2.3.5' '1.2.3.6')
sshHosts=('1.2.3.4' '1.2.3.4')
sslPathOnServer=('/etc/apache2/ssl/' '/etc/specialprocess/ssl/')
keyFileNameOnServer=('www.key' 'pub.key')
certFileNameOnServer=('www.crt' 'cert.crt')
chainFileNameOnServer=('www-chain.pem' 'chain.pem')
serverStopCommand=('/etc/init.d/apache2 stop' '/etc/init.d/specialprocess stop')
serverStartCommand=('/etc/init.d/apache2 start' '/etc/init.d/specialprocess start')

# Work is done here, no further edits!
i=0
for domain in ${domains[@]}; do
        # Ascertain if the certificate will expire in 3 weeks or less - stops lets encrypt nagging e-mails
        new=0
        if [ -d "$certPath$domain" ]
        then
                openssl x509 -in $certPath$domain/cert.pem -checkend $(( 86400 * 21 )) -noout
                result=$?
        else
                new=1
                result=1
        fi
        if [[ $result == 1 ]]
        then
                # Report the status
                if [[ $new == 1 ]]
                then
                        echo "Certificate for $domain [New]"
                        echo "Running certificate creation process..."
                else
                        echo "Ceritificate for $domain [Expiring]"
                        echo "Running renewal process..."
                fi
                # Start renewal process; create remote directory, iptables rule, start remote server
                echo -n "       Setting up iptables rules on remote server, stopping server process using certs & starting temporary micro server..."
                rule="INPUT 1 -p tcp -m tcp --dport $remotePort -d ${hosts[i]} -j ACCEPT"
                ssh root@${sshHosts[i]} << ENDSSH > /dev/null 2>&1
                eval ${serverStopCommand[i]}
                mkdir /tmp/$domain
                iptables -I $rule
                cd /tmp/$domain
                python -m SimpleHTTPServer $remotePort > /dev/null 2>&1 &
ENDSSH
                echo " Done"

                # Mount remote directory locally over SSHFS
                echo -n "       Creating mount to directory served by temporary micro server..."
                if [ -d /tmp/certbot ]
                then
                        rm -rf /tmp/certbot
                fi
                mkdir /tmp/certbot
                sshfs ${sshHosts[i]}:/tmp/$domain /tmp/certbot
                echo " Done"

                # Call letsencrypt to renew/create
                echo -n "       Call letsencrypt to renew the certificate in our store..."
                letsencrypt certonly --webroot -n -w /tmp/certbot -m $email -d $domain > /dev/null 2>&1
                echo " Done"

                # Unmount remote SSHFS directory
                echo -n "       Unmounting micro server directory and cleaning mount point..."
                umount /tmp/certbot
                rm -rf /tmp/certbot
                echo " Done"

                # SCP certificate/chain/key files to remote host
                echo -n "       Copying new certificate files to the server..."
                scp $certPath$domain/cert.pem ${sshHosts[i]}:${sslPathOnServer[i]}${certFileNameOnServer[i]} > /dev/null 2>&1
                scp $certPath$domain/privkey.pem ${sshHosts[i]}:${sslPathOnServer[i]}${keyFileNameOnServer[i]} > /dev/null 2>&1
                scp $certPath$domain/chain.pem ${sshHosts[i]}:${sslPathOnServer[i]}${chainFileNameOnServer[i]} > /dev/null 2>&1
                echo " Done"

                # Cleanup remote host; stop temp server, reload default iptables, cleanup temp directory, restart server process
                echo -n "       Stop micro server, clean up iptables rule & start server process using certs... "
                ssh ${sshHosts[i]} << ENDSSH > /dev/null 2>&1
                kill \`ps -ef | grep SimpleHTTPServer | grep -v grep | awk '{ print \$2; }'\`
                iptables -D INPUT 1
                rm -rf /tmp/$domain
                eval ${serverStartCommand[i]}
ENDSSH
                echo " Done"

                # Confirm completion
                if [[ $new == 1 ]]
                then
                        echo "Certificate for $domain [Created]"
                        echo "Creation process complete"
                else
                        echo "Renewal process complete"
                        echo "Certificate for $domain [Renewed]"
                fi
        else
                # Certificate needs no renewal, indicate as such
                echo "Certificate for $domain [OK]"
        fi
        i=$i+1
done

You’ll need to modify the variables/arrays at the start to get it working, some are obvious, but I’ll explain them anyway.

• email : This is used to create, recover and access your account. Once created, the private key for accessing your account will be stored in the let’s encrypt home directory (usully /etc/letsencrypt)
• certPath : The path to the /live directory in your let’s encrypt home directory, usually /etc/letsencrypt/live
• remotePort : Leave this at 80; it’s just here to help setup iptables rules – it cannot be changed in let’s encrypt config
• domains : An array of domains for which you want certificates
• hosts : An array of IP addresses corresponding to the aforementioned certificates – keep the ordering the same!
• sshHosts : An array of IP addresses used for administering each of the aforementioned domains; for example you may have several web servers/domains all being served from the same physical host – this would then be an array of identical IPs, corresponding in array size with the number of domains
• sslPathOnServer : What is the file system path to each domain’s certificates on the host server
• keyFileNameOnServer : Within that directory, what is the key file name
• certFileNameOnServer : Within that directory, what is the certificate file name
• chainFileNameOnServer : Within that directory, what is the chain file name
• serverStopCommand : For each domain, what command stops the server process the domain is used for
• serverStartCommand : Likewise, what is the command to start the server process again

That’s it! Best of luck if you choose to go this way and do let me know if the script is useful to you.

A small service I created for personal use had its SSL certificate expire a few weeks ago. I replaced it using my go-to free certificate provider, StartSSL. Unfortunately, accessing this service from my iPhone still didn’t work afterwards, or rather it did, but the certificate still wasn’t trusted. I was forced to ignore this issue at the time due to other pressing matters and, as the service was just used by me, on my device, I simply set the app to ignore the issue until I had time to diagnose the problem properly.

Today I replaced my e-mail signing certificate, also from StartSSL. This time I knew something was wrong; adding the certificate to Thunderbird and sending a test e-mail to be retrieved by my iPhone, the certificate again wasn’t trusted.

Turns out that silently, unbeknownst to me and I’m sure many others, the following chain of events has occurred

1. StartCom which runs StartSSL were taken over by WoSign
2. Due to past issues with WoSign regarding the erroneous issuing of certificates for a domain that they had no authority to issue for, they had their root certificate rights revoked by Apple, Mozilla, Google and others
3. Since the take over of StartSSL, the aforementioned device/software companies have also ceased to trust the root certificates of StartCom.

This has resulted in my having to spring into action. The silent take-over of a previously trusted organisation by a dubious one located in a political jurisdiction known to be questionable with respect to web security is worrying enough, but with the added pain of being unable to send signed e-mail and issue new certs for my soon to expire domains required to be addressed immediately.

I’ve found that comodo, a trusted certficate authority, will issue free e-mail signing certs so I’ve jumped over to that immediately for e-mail and that is running nicely.

In addition there is “LetsEncrypt”, which, once installed, allows you to generate trusted SSL certificates for your domains using the following command

letsencrypt certonly --manual -d my.domain.com

When running the above it’s important to remember that validation you have control over the domain for which the certificate will be issued requires you to

• Be running a web server on port 80 on the domain
• Said port 80 server to be accessible publicly through your firewall
• For you to place a file to be served on the web root of this server

You can always stop apache (or whatever your web server is) and run up a simple stand-alone python server to do the job, but you won’t be able to get away without changing your firewall rules.

This is a departure from the way I’m used to doing things which used the administrative e-mail address registered to the domain and code sent to this for validating I owned the domain. This was nicer as it meant I didn’t have to play with my config all that much (some servers I run use non-standard ports and don’t have port 80 open by default for example).

Anyhow, from my reading of the documentation about LetsEncrypt, it seems that all major browsers and Apple iOS are trusting certificates signed this way. I’ll update this post if that turns out not to be the case, but I’m optimistic.

Hopefully this post will help to raise awareness and also assist some folk out of a bit of a pickle with their security arrangements.

The other day I had cause to turn off my iPhone. Unlike many who leave theirs permanently on, it’s something I often do; restaurants, theatre, cinema, church to name but a few places where it’s simply good etiquette to do so. Anyway, I digress.

When I switched the device back on again some hours later, I was greeted with a request to “re-activate” it using my apple ID. The problem with this is that for those security concious people like myself who place their iCloud password in LastPass, you cannot get access to the the LastPass app if your phone is locked, which a de-activated phone effectively is.

Fortunately I was within easy reach of a PC on which I could install LastPass, login, retrieve the requisite password and unlock the phone.

This could have been so much worse though; with a phone that could do nothing but call 999 and no PC to retrieve my password, I could have been incommunicado while abroad or some other such significant inconvenience.

I simply didn’t *know* that an iPhone could just de-activate its self like that – certainly a risk worth evaluating when you decide what password (random, unknown, in LastPass v.s. simpler, recallable) to utilise for the purpose!

Everytime I renew my S/MIME certificate I always seem to wind having to consult the OpenSSL man pages to ascertain the approrpiate command to use in order to extract the certificate from the PKCS12 file I get from my certificate authority so I can share said certificate on my contact page. As I’m sure I’m not the only one who winds up having to look this up, I’m sharing it here.

openssl pkcs12 -in container.p12 -clcerts -nokeys -out public.pem

If you then investigate the contents of the PEM file inside a text editor you’ll find your certificate between the BEGIN and END lines – you should share this content on your website or directly with contacts so that your signed e-mails can be verified as to their origin and also so that contacts can encrypt mail such that it can only be read by you if required.