Let's Encrypt Wildcard Certificates with certbot, BIND, apache and exim

Updated 3rd January 2021.

This is a description of how to use Let's Encrypt wildcard certificates on a small home web/email server running Debian. It uses the following components:

Each website / domain will have its own wildcard certificate covering the root domain and any sub-domains. Email will be sent and received using TLS with the Let's Encrypt certificate for the domain that handles email.

Enabling Dynamic Update to BIND (RFC 2136)

When asking for a wildcard certificate, certbot pushes a record to DNS, which Let's Encrypt then retrieves to prove that you have control of the domain. This process can be fully automated if BIND is set up accept dynamic updates from certbot.

Generate a key to secure the update process:

$ cd /etc/bind
$ sudo dnssec-keygen -a HMAC-SHA512 -b 512 -n HOST certbot.

This produces a private and a public key in separate files, the files being named with the keyname, algorithm number and key fingerprint. The key files are owned by root and in the “bind” group.

Place the following configuration in the file /etc/bind/named.conf.certbot. This configuration file sets up the key which certbot will use to push updates to BIND. It also sets up a zone for the subdomain "_acme-challenge", which Let's Encrypt will query during the issuing of a wildcard certificate (as part of its DNS01 challenge). By having the subdomain in its own zone we can push updates to it without disturbing any of the entries in the parent domain. Here the text mydomain.com should be replaced with your own domain name root. If you are going to require a certificate for each of multiple domains replicate the zone clause for each domain, replacing the name of the domain in each as required.

key "certbot." {
        algorithm hmac-sha512;
        secret "private key from the file Kcertbot.+165+?????.private";
};

zone "_acme-challenge.mydomain.com" {
        type master;
        file "/var/lib/bind/db._acme-challenge.mydomain.com";
        check-names warn;
        update-policy {
                grant certbot. name _acme-challenge.mydomain.com. txt;
        };
};
The contents of /etc/bind/named.conf.certbot

The quoted text in the secret clause gets replaced with the private key in the “.private” file, generated above, making sure to retain the quotes. Be sure to include the check-names warn; clause. Without it BIND will reject the underscore in the "_acme-challenge" domain, which is a requirement of Let's Encrypt, and the DNS zone will not be started, leading to errors when one tries to update it. It is a good idea to monitor /var/log/syslog the first time BIND is restarted, so one can check the output from BIND to make sure all zones, especially those containing "_acme-challenge", are loaded.

The file should be owned by root and in the bind group It should be generally inaccessible, to protect the private key, but readable by the bind group. The commands to do this are:

$ sudo chown root:bind /etc/bind/named.conf.certbot
$ sudo chmod 640 /etc/bind/named.conf.certbot

You now need to make a zone file, /var/lib/bind/db._acme-challenge.mydomain.com, for each domain. The contents of each zone file should be as follows. We keep the file in /var rather than /etc as it will be updated by BIND and some distributions use AppArmor to restrict programs from writing to /etc.

$TTL 43200	; 12 hours
_acme-challenge.mydomain.com. IN	SOA mydomain.com. hostmaster.mydomain.com. (
				2021010205 ; serial
				28800      ; refresh (8 hours)
				7200       ; retry (2 hours)
				604800     ; expire (1 week)
				86400      ; minimum (1 day)
				)
			NS	secondary.net.
			NS	mydomain.com.
			TXT	"127.0.0.1"
The contents of /var/lib/bind/db._acme-challenge.mydomain.com

As before the text mydomain.com should be replaced with your own domain name root. The text secondary.net should be replaced with the domain name of your secondary DNS server. It is assumed that your primary DNS server is the BIND instance that we are currently working with. It is also assumed that your hostmaster email address is hostmaster#mydomain.com.

Each zone file needs to be writable by BIND. The commands to set the necessary permissions (repeat for each file) are:

$ sudo chown root:bind /var/lib/bind/db._acme-challenge.mydomain.com
$ sudo chmod 664 /var/lib/bind/db._acme-challenge.mydomain.com

We now need to delegate the _acme-challenge subdomain, for each domain, to our newly created zones. Do this by adding a line:

_acme-challenge IN NS mydomain.com.

to the zone file of each domain (typically called /etc/bind/db.mydomain.com). Remember to replace mydomain.com with your own domain name. If you are using views, the line will have to go inside the zone file for the view that is visible to the Let's Encrypt servers on the internet. For example, if you have a view for your LAN and another view for the Internet, the line will have to go in the zone file for the Internet view.

Finally we need to include out top level configuration file in the configuration file for BIND. Add the line:

       include "/etc/bind/named.conf.certbot";

to the file /etc/bind/named.conf.local. Again, if you are using views, the line will have to go inside the clause for the view that is visible to the Let's Encrypt servers on the internet.

Restart BIND to activate the changes:

$ sudo systemctl restart bind9

Testing Dynamic Update

You can skip this section, of if you are conservative you can use the techniques in this section to verify that dynamic DNS updating is working before integrating it with certbot.

Check that the configuration for BIND is correct by issuing the command:

$ sudo named-checkconf

Use the nsupdate and dig commands to check that the dynamic updating is working, by adding then deleting a record from your DNS server. To add a record:

$ sudo nsupdate -k /etc/bind/Kcertbot.+165+?????
> server mydomain.com
> update add _acme-challenge.mydomain.com 86400 TXT 192.168.1.1
> send

Kcertbot.+165+????? should be replaced with the name of the key file that you generated in the first step above, without the .private or .key suffix. The line server mydomain.com should be updated to reflect the domain of your name server. If you have separate internal and external views in your DNS system, you may need to explicitly use the external facing IP address of your network so the nsupdate command updates the external facing view. If you try to update the internal view it will fail as the _acme-challenge zone will not exist or be enabled for update in that view.

Now query the DNS server to see if it has been updated with the TXT record as requested:

$ dig @mydomain.com _acme-challenge.mydomain.com txt

Again, make sure mydomain.com is replaced with your own domain and the same caveats apply to the name server name (in the arument beginning with '@') as for the nsupdate command above. You should see the TXT record "_acme-challenge.mydomain.com 86400 TXT 192.168.1.1" appear in the answer section of the output of dig if the update is working correctly.

Now remove the record from the DNS server:

$ sudo nsupdate -k /etc/bind/Kcertbot.+165+?????
> server mydomain.com
> update delete _acme-challenge.mydomain.com 86400 TXT 192.168.1.1
> send

And use dig as before to check that the TXT record has gone.

You can do this test for each of your domains.

Set Up certbot to get Wildcard Certificates

Make sure the packages certbot, python3-certbot-apache and python3-certbot-dns-rfc2136 are installed.

Place the following configuration in the file /etc/letsencrypt/dns_rfc2136_credentials.txt. The DNS server name should be replaced with the name of your own DNS server name. If you use views in your DNS take note of the caveat about internal and external views in the previous section and make sure you use a domain name or IP address which causes certbot to access the external view of your DNS server. The text private key from the file Kcertbot.+165+?????.private should be replaced with the exact text of the secret key which was placed in the key clause in the file /etc/bind/named.conf.certbot. This time you should not use quotes.

# Target DNS server
dns_rfc2136_server = mydomain.com
# Target DNS port
dns_rfc2136_port = 53
# TSIG key name
dns_rfc2136_name = certbot.
# TSIG key secret
dns_rfc2136_secret = private key from the file Kcertbot.+165+?????.private
# TSIG key algorithm
dns_rfc2136_algorithm = HMAC-SHA512
Contents of the file /etc/letsencrypt/dns_rfc2136_credentials.txt

The file /etc/letsencrypt/dns_rfc2136_credentials.txt should be set so that only root can read it, to protect the secret key that it contains. Use the commands:

$ sudo chown root:root /etc/letsencrypt/dns_rfc2136_credentials.txt
$ sudo chmod 600 /etc/letsencrypt/dns_rfc2136_credentials.txt

Get Wildcard Certificates

You are now ready to get a wildcard certificate from Let's Encrypt. Use the command:

$ sudo /usr/bin/certbot certonly --dns-rfc2136 --dns-rfc2136-credentials /etc/letsencrypt/dns_rfc2136_credentials.txt -d mydomain.com -d *.mydomain.com

The certificate obtained will be valid for the root domain as well as all subdomains. Be sure to explicitly include the root domain as it is not included in the wildcard. If you want to do a test, without actually installing a certificate, append the option --dry-run to the above command.

Issue the above command for each of your domains, replacing the name of the domain as necessary. In this way you will end up with a wildcard certificate for each domain and its subdomains. You could put all your domains in one certificate (with a single invocation of certbot), but then there is an unnecessary connection between your domains.

Useful certbot Techniques

Each certbot certificate and its configuration is referred to by a certificate name. A list of certificates by name can be had with the command certbot certificates. When a certificate is renewed it does not get a new name and the sequence of renewed certificates form a lineage. New domains can be added to a certificate whilst retaining the lineage, but deleting a domain means a new name and lineage will be started. A new lineage will also be started if you shift from using single domains on a certificate to using a wildcard. The new lineage is typically the domain name followed by a number.

After a while you can get unwanted and unused lineages for a domain. If you want to remove them the easiest way is to revoke and delete all the existing certificates / lineages for the domain and start again. To revoke and delete a certificate use:

$ sudo certbot revoke --cert-name mydomain.com-0001 --delete-after-revoke

Here "mydomain.com-0001" is the name of the certificate as shown by the certbot certificates command. If you delete all the lineages for a domain the first new lineage (created with certbot certonly ...) will be named with just the domain name with no numeric suffix.

Using the Certificate with exim

Apart from securing your web server, your Let's Encrypt certificate can be used to secure your exim based email server. In this case, you want the domain of your email server to be included in the certificate. For a small home based server, with the email domain the same as the web domain, the same certificate can be used for apache and exim. All that is need is to automatically copy the certificate into exim's directory, taking care that the permissions are set so exim can read it but nothing else can.

To this end one can make a script that renews the keys using certbot then copies the certificate across to exim. Create a file /etc/letsencrypt/le-renew with the following contents. This file should be readable by root.

cd /etc/letsencrypt
cp live/mydomain.com/fullchain.pem live/mydomain.com/exim.crt
cp live/mydomain.com/privkey.pem live/mydomain.com/exim.key
chmod 640 live/mydomain.com/exim.crt live/mydomain.com/exim.key
chgrp Debian-exim live/mydomain.com/exim.crt live/mydomain.com/exim.key
mv live/mydomain.com/exim.crt /etc/exim4
mv live/mydomain.com/exim.key /etc/exim4
Contents of the file /etc/letsencrypt/le-renew

Run the script /etc/letsencrypt/le-renew to renew certificates and automatically update exim's certificate. You can test whether exim is sending and receiving using TLS by looking at the full headers for emails sent to and received from another email server on the Internet, or by using a site such as https://www.checktls.com/.

In theory the copying of the certificate to exim could be done as a "deploy hook" in certbot, but deployment hooks don't seem to work.

Automatic Certificate Renewal

To automatically renew your certificates, make a cron job for root which periodically runs the script /etc/letsencrypt/le-renew. Running it weekly should suffice for continuity of your certificates, as certbot renews any certificates that renew within the next 30 days. A sample crontab entry to renew at 3:23am every Tuesday would be:

23 3 * * 2 /etc/letsencrypt/le-renew


Version control information: $Id$