Getting Envoy to pick up rotated certificates
Like many people, I use cert-manager to automatically
renew my website’s TLS certificates with Let’s Encrypt. Unlike many
people, I don’t use an Ingress controller to get traffic into my cluster, I just have a few
instances of Envoy that terminate TLS and route traffic to the appropriate
backend. Cert-manager handles the mechanics of certificate renewal very efficiently; it runs a
controller loop that checks all my Certificate objects for expiration, and when a certificate is
close to expiring, it goes out and renews it. It then updates a Kubernetes Secret with the new key
material, and Kubernetes then makes that new data available to pods that have mounted the Secret as
a volume. From there, it’s up to the application to notice that some symlinks have moved around and
reload the certificate. Up until very recently, Envoy did not bother to check. So at some point, you
had to do a rolling restart of the Envoy deployment to pick up the new certificate. Because there is
30 days between when cert-manager renews the certificate and when the old certificate actually
expired, this was rarely a problem in practice. If any of your machines went down, or you edited the
config file to add a new route, or you upgraded Envoy itself, the pod containing Envoy would
restart, pick up the new certificates, and you’d never notice that it wasn’t automatically reloading
the certificate.
As someone who doesn’t like to leave important production operations to chance, though, I knew I needed a better system. Fortunately, Envoy added a way to reload certificates with the 1.14 release. Let’s try that.
The mechanics⌗
Everyone’s Envoy configuration is different, so I’m just going to provide a very minimal
envoy.yaml that we’ll modify to make certs automatically reload. You can then apply this to your
own configuration. (You can experiment with this on your workstation by building Envoy, or
extracting the binary from a Docker image with docker cp. That’s what I do for all my local Envoy
work. Although Envoy does not distribute binaries, the binary from the Docker image works great on
my Ubuntu 19.10 workstation.)
Here is a basic envoy.yaml that serves HTTPS on port 10000 with a static response:
static_resources:
listeners:
- name: test
address:
socket_address:
protocol: TCP
address: 127.0.0.1
port_value: 10000
listener_filters:
- name: "envoy.listener.tls_inspector"
typed_config: {}
filter_chains:
- tls_context:
common_tls_context:
alpn_protocols: ["h2", "http/1.1"]
tls_certificates:
- certificate_chain:
filename: "/certs/tls.crt"
private_key:
filename: "/certs/tls.key"
filters:
- name: envoy.http_connection_manager
typed_config:
"@type": type.googleapis.com/envoy.config.filter.network.http_connection_manager.v2.HttpConnectionManager
stat_prefix: test
route_config:
virtual_hosts:
- name: test
domains: ["*"]
routes:
- match: { prefix: "/" }
direct_response:
status: 200
body:
inline_string: "Hello from Envoy"
http_filters:
- name: envoy.router
Put your TLS key and cert in /certs, and curl https://localhost:10000/ will return “Hello from
Envoy”. It works. You can do whatever you want to /certs and Envoy will keep using the TLS
configuration that it loaded at startup.
To fix that, we need to make the TLS context for our listener use SDS instead of a static configuration.
The first step is to create another config file that contains information about the secret
discovery. I put my main envoy.yaml in a ConfigMap that gets mounted into /etc/envoy, and just
added a sds.yaml to that ConfigMap to store the SDS configuration. All it is is a plaintext
representation of what an SDS API server would serve to Envoy, if it was getting configuration from
an xDS server and not the filesystem. It looks like:
resources:
- "@type": "type.googleapis.com/envoy.api.v2.auth.Secret"
tls_certificate:
certificate_chain:
filename: "/certs/tls.crt"
private_key:
filename: "/certs/tls.key"
While this looks almost exactly what we put in the main envoy.yaml before, this is what triggers
the code to start watching various directories for changes with inotify and lead to the eventual
refreshing of your certificate.
We also need to make some changes to envoy.yaml itself. Instead of statically configuring the
listener with a certificate, we need the listener to load the certificate from SDS. In the
listener’s filter_chains section, we’ll change the tls_context to a more general
transport_socket, and then point it at our sds.yaml. (It is not necessary to convert
tls_context to transport_socket, but tls_context will be gone by the end of the year, so you
might as well change it now.)
So now instead of:
listeners:
- name: test
...
filter_chains:
- tls_context: {...}
filters: [...]
We’ll have:
listeners:
- name: test
...
filter_chains:
- transport_socket:
name: "envoy.transport_sockets.tls"
typed_config:
"@type": "type.googleapis.com/envoy.api.v2.auth.DownstreamTlsContext"
common_tls_context:
alpn_protocols: ["h2", "http/1.1"]
tls_certificate_sds_secret_configs:
sds_config:
path: /etc/envoy/sds.yaml
filters: [...]
Using SDS also activates other parts of Envoy’s code that wants Envoy to have some identifying
information associated with the node. You can supply that on the command line, or in the bootstrap
config with a node configuration at the top level:
node:
id: test
cluster: test
If you omit this, you’ll get an error like:
TlsCertificateSdsApi: node 'id' and 'cluster' are required. Set it either in 'node' config or via --service-node and --service-cluster options.
(In production, I use the pod’s hostname, like envoy-b958c94b7-2fbws, for the ID and
ingress:public:https as the cluster name. That is what my
cluster discovery service calls my cluster. It doesn’t matter
for this, but it does matter for other things. You probably already have this set up.)
The result is a final envoy.yaml that looks like:
node:
id: test
cluster: test
static_resources:
listeners:
- name: test
address:
socket_address:
protocol: TCP
address: 127.0.0.1
port_value: 10000
listener_filters:
- name: envoy.listener.tls_inspector
typed_config: {}
filter_chains:
- transport_socket:
name: envoy.transport_sockets.tls
typed_config:
"@type": type.googleapis.com/envoy.api.v2.auth.DownstreamTlsContext
common_tls_context:
alpn_protocols: ["h2", "http/1.1"]
tls_certificate_sds_secret_configs:
sds_config:
path: /etc/envoy/sds.yaml
filters:
- name: envoy.http_connection_manager
typed_config:
"@type": type.googleapis.com/envoy.config.filter.network.http_connection_manager.v2.HttpConnectionManager
stat_prefix: test
route_config:
virtual_hosts:
- name: test
domains: ["*"]
routes:
- match: { prefix: "/" }
direct_response:
status: 200
body:
inline_string: "Hello from Envoy"
http_filters:
- name: envoy.router
With that running, your certificates should be used by Envoy as soon as they are rotated!
There is a delay between a secret being updated and the volume mount changing, controlled by your cluster administrator (it’s a parameter to the Kubelet) – if you are watching the Kubernetes event log or cert-manager’s log, you might not see the new certificate as soon as you think it’s ready, but it should be available on the order of 5 minutes later.
Envoy also prints some logs at the debug level:
[2020-04-26 18:41:04.243][23137][debug][file] [source/common/filesystem/inotify/watcher_impl.cc:72] notification: fd: 1 mask: 80 file: ..data
[2020-04-26 18:41:04.243][23137][debug][file] [source/common/filesystem/inotify/watcher_impl.cc:88] matched callback: directory: ..data
[2020-04-26 18:41:04.243][23137][debug][config] [source/extensions/transport_sockets/tls/ssl_socket.cc:678] Secret is updated.
[2020-04-26 18:41:04.245][23137][debug][file] [source/common/filesystem/inotify/watcher_impl.cc:88] matched callback: directory: ..data
Be aware that debug logging is not on by default; you’ll have to turn it on if you want to watch
this happen the first time. In general, the way that I check that it worked is by looking at the
/certs admin API endpoint, or by the server.days_until_first_cert_expiring stat (which you
should be feeding into your monitoring).
The details⌗
When I read the changelog for Envoy 1.14, I knew I wanted to try this feature, but I also assumed that it wouldn’t be a simple cut-n-paste job to get it working. In retrospect, I was wrong; it was actually simple to get working since it was designed to exactly work with Kubernetes’s filesystem structure, and I happen to deploy on Kubernetes. I wrote a little test program, to try things out on my workstation before blindly forging ahead in production. This ended up taking quite a bit of time and did not work initially.
The first version of my code assumed that the atomic updating would work like the rest of Envoy
(through its Runtime
configuration) – i.e., put your certificates in some directory, and symlink another directory (call
it data) to that. Your certs are then in /whatever/data/tls.key and /whatever/data/tls.crt,
and /whatever/data is just a symlink to /somewhere/20190401-certs. When you want the
certificates to rotate, you symlink the new directory to .tmp or something, and then atomically
replace data with .tmp, mv -Tf .tmp data.
However, Envoy does not recognize that sequence for certificates. It requires you to do the exact
dance that Kubernetes does, which involves two levels of symlinks. If you have a volume mount
/certs, then the current version of your Kubernetes secret is actually stored in
/certs/..timestamp (where timestamp is actually something like 2020_04_09_17_25_30.145602340).
So you’ll have /certs/..timestamp/tls.key, etc., as a normal file. This current timestamp is then
linked to a directory called ..data. Finally, /certs/tls.key (and friends), are linked to
..data/tls.key. When a data update arrives, the files are written to a new ..timestamp
directory, and the ..data symlink is atomically replaced. This is close to what I did in the first
version of my program, but not exactly the same. As a result, Envoy did not notice any changes my
program made. I changed my program to do exactly what Kubernetes does, and then it started working.
Now that program exists so you can test locally without having to understand the details ;)
Here is the ls output of that sort of directory structure:
# ls -laR jrock.us
jrock.us:
total 4
drwxrwxrwt 3 root root 140 Apr 9 17:25 .
drwxr-xr-x 1 root root 4096 Apr 9 17:25 ..
drwxr-xr-x 2 root root 100 Apr 9 17:25 ..2020_04_09_17_25_30.145602340
lrwxrwxrwx 1 root root 31 Apr 9 17:25 ..data -> ..2020_04_09_17_25_30.145602340
lrwxrwxrwx 1 root root 13 Apr 9 17:25 ca.crt -> ..data/ca.crt
lrwxrwxrwx 1 root root 14 Apr 9 17:25 tls.crt -> ..data/tls.crt
lrwxrwxrwx 1 root root 14 Apr 9 17:25 tls.key -> ..data/tls.key
jrock.us/..2020_04_09_17_25_30.145602340:
total 8
drwxr-xr-x 2 root root 100 Apr 9 17:25 .
drwxrwxrwt 3 root root 140 Apr 9 17:25 ..
-rw-r--r-- 1 root root 0 Apr 9 17:25 ca.crt
-rw-r--r-- 1 root root 3558 Apr 9 17:25 tls.crt
-rw-r--r-- 1 root root 1679 Apr 9 17:25 tls.key
Looking at the tests in the PR where this
feature was introduced was helpful, as is the
related ticket. (It didn’t make sense to me until
I just kubectl exec'd into a container and ran ls -laR on a mounted secret, though. If you are
implementing something similar, I recommend doing that. I would also greatly appreciate a link to
the code in Kubernetes that manages this. I spent about 20 minutes looking, but couldn’t find it,
which annoys me.)
Conclusion⌗
In the end, this was a very simple change. Here’s all I needed to do for my own personal site: jrock.us#3d986…
Anyway, I hope this is helpful to someone. I am glad I no longer have to care about certificates, and hopefully you don’t have to either!