How to launch nginx-ingress and cert-manager 0.6.2 in Kubernetes using AWS DNS Route53 Validation

At my job (in Fravega) we have been struggling for some days dealing with certificates. It turns out that we didn’t find an updated guide for this, so we decided to write our own.

Following this guide but with changes required for dns validation(needed for private ingress!) and to work with cert manager v0.6.2

This guide assumes that you have K8s cluster working with external dns and nginx-ingress-controller installed, the following steps are:

  1. Install helm
  2. Install cert manager
  3. Create a user in AWS with route53 permissions
  4. Create Staging ClusterIssuer with DNS validation
  5. Create Certificate
  6. Create a test Ingress

Install helm

Helm is a package manager for Kubernetes. It allows you to install packages of pre-configured Kubernetes resources and publish them as charts.

If you have already installed and setup kubectl to access your cluster, you can easily install helm following this instruction.

Install Cert Manager

The component which is responsible for obtaining and renewing certificates is cert-manager.

Its helm chart is not yet in a stable repo, so I recommend launching it from Github:

# Install the cert-manager CRDs. We must do this before installing the Helm
# chart in the next step
$ kubectl apply -f https://raw.githubusercontent.com/jetstack/cert-manager/release-0.6/deploy/manifests/00-crds.yaml
# Create and label ns for cert-manager
$ kubectl create namespace cert-manager && kubectl label namespace cert-manager certmanager.k8s.io/disable-validation=true

# Install cert-manager
$ helm install --name cert-manager --namespace cert-manager stable/cert-manager

Usingcert-manager we set up a ClusterIssuer so that we can obtain TLS certificates.

The Certificate entity keeps information used to verify certificates for the current ClusterIssuer (Certificate Authority).
cert-manager communicates via ClusterIssuer (i.e. with Let’s Encrypt) to obtain a certificate and then creates a Secret in K8s, containting the generated certificate. These certificates will be used by the Ingress.

Create a user in AWS with route53 permissions

Cert-manager requires the following IAM policy.

{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": "route53:GetChange",
"Resource": "arn:aws:route53:::change/*"
},
{
"Effect": "Allow",
"Action": "route53:ChangeResourceRecordSets",
"Resource": "arn:aws:route53:::hostedzone/*"
},
{
"Effect": "Allow",
"Action": "route53:ListHostedZonesByName",
"Resource": "arn:aws:route53:::hostedzone/*"
}
]
}

Create Staging ClusterIssuer with DNS validation

Let’s Encrypt has two environments — staging and production. The staging environment issues certificates signed by ‘fake’ CAs, and has extended rate limits, so to ensure that everything works fine (and allow for some failures), I recommend using Let’s Encrypt staging api at first.

The difference between just Issuers and ClusterIssuers is that the latter are created on a cluster lever, not namespace level.

Create a file letsencrypt-staging-dns.yaml with the following data:

apiVersion: certmanager.k8s.io/v1alpha1
kind: ClusterIssuer
metadata:
name: letsencrypt-staging-dns
spec:
acme:
# The ACME server URL
server: https://acme-staging-v02.api.letsencrypt.org/directory
# Email address used for ACME registration
email: "certificates@example.com"
# Name of a secret used to store the ACME account private key
privateKeySecretRef:
name: letsencrypt-staging-dns
# Enable the DNS-01 challenge provider
dns01:
providers:
- name: dns
route53:
region: us-east-1
accessKeyID: YOURACCESSKEYID
secretAccessKeySecretRef:
name: acme-route53
key: secret-access-key

Create a text file with the secret-access-key and generate the secret:

kubectl create secret generic acme-route53 -n=kube-system --from-file=secret-access-key

Now run kubectl create -f letsencrypt-staging-dns.yaml, to create the ClusterIssuer.

You can verify that it’s created using kubectl get clusterissuers

NAME             AGE
letsencrypt-staging-dns 1m

You can also use describe instead of get to display more info and actual status.

Create Certificate

Create a file nginx.sandbox.cluster.com.yaml with the following data:

apiVersion: certmanager.k8s.io/v1alpha1
kind: Certificate
metadata:
name: nginx-sandbox-cluster-com
namespace: cert-manager
spec:
secretName: nginx-sandbox-cluster-com-tls
issuerRef:
name: letsencrypt-staging-dns
kind: ClusterIssuer
commonName: nginx.sandbox.cluster.com
dnsNames:
- nginx.sandbox.cluster.com
acme:
config:
- dns01:
provider: dns
domains:
- nginx.sandbox.cluster.com

Create a test Ingress

Launch nginx in the default namespace and create a service for it.

Run kubectl run nginx --image nginx
and kubectl expose deploy nginx --port 80

You’ve just launched a pod with nginx in the default namespace with a service called nginx.

Create ingress.yaml

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
annotations:
ingress.kubernetes.io/ssl-redirect: "false"
kubernetes.io/ingress.class: nginx
labels:
app: nginx
name: nginx-sandbox-cluster-com
namespace: cert-manager
spec:
tls:
- hosts:
- nginx.sandbox.cluster.com
secretName: nginx-sandbox-cluster-com-tls
rules:
- host: nginx.sandbox.cluster.com
http:
paths:
- backend:
serviceName: nginx
servicePort: http
path: /

Run kubectl create -f ingress.yaml - we’ve just created the Ingress.
The certificate should be ready in about 30 seconds.

Now the whole traffic goes through https, and we’ve got certificates from Let’s Encrypt.

Move to Production

You will only need to change Letsencrypt endpoint from :

https:acme-staging-v02.api.letsencryp.org/directory to https:acme-v02.api.letsencryp.org/directory

Then change the name of the ClusterIssuer from:

letsencrypt-staging-dns to letsencrypt-production-dns

You can verify the cert using curl -v https://nginx.sandbox.cluster.com

*   Trying XX.XX.XX.XX...
* Connected to nginx.sandbox.cluster.com (XX.XX.XX.XX) port 443 (#0)
* found 148 certificates in /etc/ssl/certs/ca-certificates.crt
* found 598 certificates in /etc/ssl/certs
* ALPN, offering http/1.1
* SSL connection using TLS1.2 / ECDHE_RSA_AES_256_GCM_SHA384
* server certificate verification OK
* server certificate status verification SKIPPED
* common name: nginx.sandbox.cluster.com (matched)
* server certificate expiration date OK
* server certificate activation date OK
* certificate public key: RSA
* certificate version: #3
* subject: CN=nginx.sandbox.cluster.com
* start date: Thu, 31 May 2018 16:45:27 GMT
* expire date: Wed, 29 Aug 2018 16:45:27 GMT
* issuer: C=US,O=Let's Encrypt,CN=Let's Encrypt Authority X3
* compression: NULL
* ALPN, server accepted to use http/1.1
> GET / HTTP/1.1
> Host: nginx.sandbox.cluster.com
> User-Agent: curl/7.47.0
> Accept: */*
>
< HTTP/1.1 503 Service Temporarily Unavailable
< Server: nginx/1.13.9
< Date: Thu, 31 May 2018 17:54:11 GMT
< Content-Type: text/html
< Content-Length: 213
< Connection: keep-alive
<
<html>
<head><title>503 Service Temporarily Unavailable</title></head>
<body bgcolor="white">
<center><h1>503 Service Temporarily Unavailable</h1></center>
<hr><center>nginx/1.13.9</center>
</body>
</html>
* Connection #0 to host nginx.sandbox.cluster.com left intact

That’s all :)

[OPTIONAL] Looking at the logs

You can use some tool like kubetail to get the logs and see what’s happening!

kubetail cert-manager -n cert-manager

[cert-manager-cert-manager-7b76495cbf-zcvgv] I0531 14:28:07.143588       1 sync.go:246] Issuing certificate... 
[cert-manager-cert-manager-7b76495cbf-zcvgv] I0531 14:28:07.143633 1 acme.go:159] getting private key (letsencryptdns->tls.key) for acme issuer kube-system/letsencryptdns
[cert-manager-cert-manager-7b76495cbf-zcvgv] I0531 14:28:07.143982 1 logger.go:27] Calling GetOrder
[cert-manager-cert-manager-7b76495cbf-zcvgv] I0531 14:28:07.753839 1 logger.go:37] Calling FinalizeOrder
[cert-manager-cert-manager-7b76495cbf-zcvgv] I0531 14:28:08.526455 1 issue.go:101] successfully obtained certificate: cn="nginx.sandbox.cluster.com" altNames=[nginx.sandbox.cluster.com] url="https://acme-staging-v02.api.letsencrypt.org/acme/order/6189521/1555676"
[cert-manager-cert-manager-7b76495cbf-zcvgv] I0531 14:28:08.539450 1 sync.go:265] Certificate issued successfully
[cert-manager-cert-manager-7b76495cbf-zcvgv] I0531 14:28:08.539511 1 helpers.go:162] Found status change for Certificate "nginx-sandbox-cluster-com" condition "Ready": "False" -> "True"; setting lastTransitionTime to 2018-05-31 14:28:08.539505046 +0000 UTC m=+1424.707831578
[cert-manager-cert-manager-7b76495cbf-zcvgv] I0531 14:28:08.546779 1 sync.go:177] Certificate default/nginx-sandbox-cluster-com scheduled for renewal in 1438 hours
[cert-manager-cert-manager-7b76495cbf-zcvgv] I0531 14:28:08.546859 1 controller.go:191] certificates controller: Finished processing work item "default/nginx-sandbox-cluster-com"

So, that’s it. I hope you find it useful and the steps are easy to follow. Please comment and/or ask questions if I missed something — I’d love to get your feedback.

Devops K8s and some random stuff from here and there. Berlin based, working at Sumup

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store