Troubleshooting Problems with Let's Encrypt Certificates

Introduction

In this guide, we'll explore troubleshooting techniques to identify the root cause if your Let's Encrypt certificate issuance or renewal encounters issues.

login to Azure

Before delving into troubleshooting, ensure you are logged into the correct Azure subscription using Visual Studio Code:

# Login to Azure
az login 

# Shows current Azure subscription
az account show --output table

# Lists all available Azure subscriptions
az account list --output table

# Sets Azure subscription to desired subscription using ID
az account set -s "anji.keesari"

Connect to the AKS Cluster

To interact with your Azure Kubernetes Service (AKS) cluster, you need to establish a connection. Depending on your role, you can use either the User or Admin credentials:

# Azure Kubernetes Service Cluster User Role
az aks get-credentials -g "rg-aks-dev" -n "aks-cluster1-dev"

# Azure Kubernetes Service Cluster Admin Role
az aks get-credentials -g "rg-aks-dev" -n "aks-cluster1-dev" --admin

# verify the aks connection by running following commands
kubectl get no
kubectl get namespace -A

Certificate Management with cert-manager

The troubleshooting process involves checking various resources within the cert-manager.

There are several resources that are involved in requesting a certificate.

Ingress -> 
    Certificate ->
                  CertificateRequest 
ACME ->
    Order  -> 
            Challenge  ->

The cert-manager flow all starts at a Certificate resource, you can create this yourself or your Ingress resource will to this for you

Step-1: Checking the Certificate resource

Start by verifying the Certificate resource in the specified namespace (e.g., argocd):

kubectl get certificate -n argocd

# output
NAME         READY   SECRET       AGE
tls-secret   True    tls-secret   85d

Look for the status of the certificates. If "Ready" is false, obtain more details:

kubectl describe certificate tls-secret -n argocd

# output
Name:         tls-secret
Namespace:    argocd
Labels:       <none>
Annotations:  <none>
API Version:  cert-manager.io/v1
Kind:         Certificate
Metadata:
.
.
.
Spec:
  Dns Names:
      argocd.yourdomain.com
  Issuer Ref:
    Group:      cert-manager.io
    Kind:       ClusterIssuer
    Name:       letsencrypt
  Secret Name:  tls-secret
  Usages:
    digital signature
    key encipherment
Status:
  Conditions:
    Last Transition Time:  2023-08-29T22:33:37Z
    Message:               Certificate is up to date and has not expired
    Observed Generation:   1
    Reason:                Ready
    Status:                True
    Type:                  Ready
  Not After:               2024-01-26T20:34:10Z
  Not Before:              2023-10-28T20:34:11Z
  Renewal Time:            2023-12-27T20:34:10Z
  Revision:                2
Events:                    <none>

Step-2: Checking the CertificateRequest

The CertificateRequest resource represents a Certificate Signing Request (CSR) in cert-manager:

kubectl get certificaterequest -n argocd

# output
NAME               APPROVED   DENIED   READY   ISSUER        REQUESTOR                                         AGE
tls-secret-gvppb   True                True    letsencrypt   system:serviceaccount:cert-manager:cert-manager   85d
tls-secret-px228   True                True    letsencrypt   system:serviceaccount:cert-manager:cert-manager   25d

Get more information about a specific CertificateRequest:

kubectl describe certificaterequest tls-secret-px228 -n argocd

# output
Name:         tls-secret-px228
Namespace:    argocd
Labels:       <none>
Annotations:  cert-manager.io/certificate-name: tls-secret
              cert-manager.io/certificate-revision: 2
              cert-manager.io/private-key-secret-name: tls-secret-ctl62
API Version:  cert-manager.io/v1
Kind:         CertificateRequest
Metadata:
 .
 .
 .
Spec:
  Extra:
    authentication.kubernetes.io/pod-name:
      cert-manager-85f68f95f-928hm
    authentication.kubernetes.io/pod-uid:
      562560ea-32ab-4294-8eab-be3eca845dfa
  Groups:
    system:serviceaccounts
    system:serviceaccounts:cert-manager
    system:authenticated
  Issuer Ref:
    Group:  cert-manager.io
    Kind:   ClusterIssuer
    Name:   letsencrypt
  Request:  LS0tLS1CRUdJTiBDRVJUSUZJQ0FURSBSRVFVRVNULS0tLS0KTUlJQ2pUQ0NBWFVDQVFBd0FEQ0NBU0l3RFFZSktvWklodmNOQVFFQkJRQURnZ0VQQURDQ0FRb0NnZ0VCQUxyUQpkajJ5M3JGcUlxNGxiQXVKNCtWMCtnWHlEd1BHaUpPUXdoNG9NS00zOFJXZ2UzMHhqR1AwUnRVNEQ3Z2VxUFh5Ck5nVDR3cHphREU2UWk0S0t3endwZGV4MHovMUJQeWxtTlo3Um9GbDNJUTZkR2l2MTNLekFhK1BJbkdjK052N1EKZ0I5bDc0QTRBYWxWeTBmV3NITzNIcEI2RGJPRTUvbmRDOHJmTEhlMmE1SXdFR09pRGhoNFdrVjNmVVRvYkc3bwpOWEoydlhPZmJFekY2T3pPL1VTeEExSTBwUENkUTdKbUVQYytrTGw4WXlJczJKcFJNa0dybTF5TGpSV1hpVXFnCmVVY2NDck8yZVliUHdKbkJUckFrNjgwekVwTm5JV2QwTGlDWDIyaUFCSFY4OS9HTzlhM1crMWFaR3M0S0E5dXIKZ0VQem1ZSmVrblhzUlIxamdJTUNBd0VBQWFCSU1FWUdDU3FHU0liM0RRRUpEakU1TURjd0tBWURWUjBSQkNFdwpINElkWkdWMkxtRnlaMjlqWkM1bGQyMHpMbUZ6YzJWMGJXRnlheTV1WlhRd0N3WURWUjBQQkFRREFnV2dNQTBHCkNTcUdTSWIzRFFFQkN3VUFBNElCQVFBL3lPcVZNdWIrazZHUzNnek5PcmQrT3ZGQ213Q2RWZ3g4NCtGT0gwVlUKb1luTTVSUWcrZmJyOUI2eDVHRzA0WG9mTVl5eUJJenZVQzd5QUg3djN4UkhOZGJVeVNteWp0NEpGVWFQcDJlcwpuSXdNMm9pSklxbEM4UEFyeG1PYlpTMkZWWTZrcTVVVXYvVHV6VTZHaUd0aTBCSmcrTFhySTQ2U2xrTHM1OEdHCm5hUVJubGRpTW51ZlN3b1VFVG1CcW5OMm0zTC94MmxBRTlTQjZFN0ZVZk9FNlI3SXBVRVpyN3doTFFwRno0VjQKQ3RmNmJaYU1tMmJiOVh6NkRQK2VaSW9obTRUc08vQ3B3ZUhkVVFoM3NCQ0l5MmNlRnk0VFhkL1UrYUptL1dXWgorelRlUHNSNnA1KzJQQ2xGWGM4bnhXR284RmxvT0xNZjVJNmRRS1FHQlpmTgotLS0tLUVORCBDRVJUSUZJQ0FURSBSRVFVRVNULS0tLS0K
  UID:      da8c7f37-8646-44ad-a532-a780eef32c43
  Usages:
    digital signature
    key encipherment
  Username:  system:serviceaccount:cert-manager:cert-manager
Status:
  Certificate:  LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUZCekNDQSsrZ0F3SUJBZ0lTQThJZ01hMWc0eXBuaU9Ud2ZVMUh0ZW1TTUEwR0NTcUdTSWIzRFFFQkN3VUEKTURJeEN6QUpCZ05WQkFZVEFsVlRNUll3RkFZRFZRUUtFdzFNWlhRbmN5QkZibU55ZVhCME1Rc3dDUVlEVlFRRApFd0pTTXpBZUZ3MHlNekV3TWpneU1ETTBNVEZhRncweU5EQXhNall5TURNME1UQmFNQ2d4SmpBa0JnTlZCQU1UCkhXUmxkaTVoY21kdlkyUXVaWGR0TXk1aGMzTmxkRzFoY21zdWJtVjBNSUlCSWpBTkJna3Foa2lHOXcwQkFRRUYKQUFPQ0FROEFNSUlCQ2dLQ0FRRUF1dEIyUGJMZXNXb2lyaVZzQzRuajVYVDZCZklQQThhSWs1RENIaWd3b3pmeApGYUI3ZlRHTVkvUkcxVGdQdUI2bzlmSTJCUGpDbk5vTVRwQ0xnb3JEUENsMTdIVFAvVUUvS1dZMW50R2dXWGNoCkRwMGFLL1hjck1CcjQ4aWNaejQyL3RDQUgyWHZnRGdCcVZYTFI5YXdjN2Nla0hvTnM0VG4rZDBMeXQ4c2Q3WnIKa2pBUVk2SU9HSGhhUlhkOVJPaHNidWcxY25hOWM1OXNUTVhvN003OVJMRURValNrOEoxRHNtWVE5ejZRdVh4agpJaXpZbWxFeVFhdWJYSXVORlplSlNxQjVSeHdLczdaNWhzL0FtY0ZPc0NUcnpUTVNrMmNoWjNRdUlKZmJhSUFFCmRYejM4WTcxcmRiN1Zwa2F6Z29EMjZ1QVEvT1pnbDZTZGV4RkhXT0Fnd0lEQVFBQm80SUNIekNDQWhzd0RnWUQKVlIwUEFRSC9CQVFEQWdXZ01CMEdBMVVkSlFRV01CUUdDQ3NHQVFVRkJ3TUJCZ2dyQmdFRkJRY0RBakFNQmdOVgpIUk1CQWY4RUFqQUFNQjBHQTFVZERnUVdCQlROV2xpay8remRSUmFqbGtEZGxtVCtSbVJTOWpBZkJnTlZIU01FCkdEQVdnQlFVTHJNWHQxaFd5NjVRQ1VEbUg2K2RpeFRDeGpCVkJnZ3JCZ0VGQlFjQkFRUkpNRWN3SVFZSUt3WUIKQlFVSE1BR0dGV2gwZEhBNkx5OXlNeTV2TG14bGJtTnlMbTl5WnpBaUJnZ3JCZ0VGQlFjd0FvWVdhSFIwY0RvdgpMM0l6TG1rdWJHVnVZM0l1YjNKbkx6QW9CZ05WSFJFRUlUQWZnaDFrWlhZdVlYSm5iMk5rTG1WM2JUTXVZWE56ClpYUnRZWEpyTG01bGREQVRCZ05WSFNBRUREQUtNQWdHQm1lQkRBRUNBVENDQVFRR0Npc0dBUVFCMW5rQ0JBSUUKZ2ZVRWdmSUE4QUIyQUR0VGQzVStMYm1BVG9zd1d3YitRRHRuMkUvRDlNZTlBQTB0Y20vaCt0UVhBQUFCaTNnMQplQm9BQUFRREFFY3dSUUloQUpSM2hUY0ltTTRZUnZvT1FpK2NRS0tLS0tCg==
  Conditions:
    Last Transition Time:  2023-10-28T21:33:36Z
    Message:               Certificate request has been approved by cert-manager.io
    Reason:                cert-manager.io
    Status:                True
    Type:                  Approved
    Last Transition Time:  2023-10-28T21:34:12Z
    Message:               Certificate fetched from issuer successfully
    Reason:                Issued
    Status:                True
    Type:                  Ready

Step-3: ACME Troubleshooting

ACME (e.g. Let's Encrypt) issuers have 2 additional resources inside cert-manager: Orders and Challenges

When requesting ACME certificates, cert-manager will create Order and Challenges to complete the request.

Step-4: Troubleshooting (Cluster)Issuers

Ensure that the (Cluster)Issuer you're using is in a ready state

kubectl get clusterissuer
kubectl get clusterissuer -n argocd 

# output
NAME          READY   AGE
letsencrypt   True    312d

If "Ready" is false, check the status:

kubectl describe clusterissuer letsencrypt
kubectl describe clusterissuer letsencrypt -n argocd 

# output
Name:         letsencrypt
Namespace:
Labels:       <none>
Annotations:  <none>
API Version:  cert-manager.io/v1
Kind:         ClusterIssuer
Metadata:
 .
 .
 .
Spec:
  Acme:
    Email:            anji.keesari@assetmark.com
    Preferred Chain:
    Private Key Secret Ref:
      Name:  letsencrypt
    Server:  https://acme-v02.api.letsencrypt.org/directory
    Solvers:
      http01:
        Ingress:
          Class:  azure/application-gateway
          Pod Template:
            Metadata:
            Spec:
              Node Selector:
                kubernetes.io/os:  linux
Status:
  Acme:
    Last Registered Email:  anji.keesari@assetmark.com
    Uri:                    https://acme-v02.api.letsencrypt.org/acme/acct/819289237
  Conditions:
    Last Transition Time:  2023-01-14T22:43:04Z
    Message:               The ACME account was registered with the ACME server
    Observed Generation:   1
    Reason:                ACMEAccountRegistered
    Status:                True
    Type:                  Ready
Events:                    <none>

Step-5: Troubleshooting Orders

When we run a describe on the CertificateRequest resource we see that an Order that has been created:

kubectl get certificaterequest -n argocd

Check the status of Orders created during the certificate request:

kubectl get order -A
kubectl get order -n argocd

# output
NAME                        STATE   AGE
tls-secret-gvppb-89069991   valid   85d
tls-secret-px228-89069991   valid   25d

Orders are a request to an ACME instance to issue a certificate. By running kubectl describe order on a particular order, information can be gleaned about failures in the process:

Once an Order is successful, you should see an event like the following:

 kubectl describe order tls-secret-px228-89069991 -n argocd

# output
Name:         tls-secret-px228-89069991
Namespace:    argocd
Labels:       <none>
Annotations:  cert-manager.io/certificate-name: tls-secret
              cert-manager.io/certificate-revision: 2
              cert-manager.io/private-key-secret-name: tls-secret-ctl62
API Version:  acme.cert-manager.io/v1
Kind:         Order
Metadata:
 .
 .
 .
Spec:
  Dns Names:
    argocd.yourdomain.com
  Issuer Ref:
    Group:  cert-manager.io
    Kind:   ClusterIssuer
    Name:   letsencrypt
  Request:  LS0tLS1CRUdJTiBDRVJUSUZJQ0FURSBSRVFVRVNULS0tLS0KTUlJQ2pUQ0NBWFVDQVFBd0FEQ0NBU0l3RFFZSktvWklodmNOQVFFQkJRQURnZ0VQQURDQ0FRb0NnZ0VCQUxyUQpkajJ5M3JGcUlxNGxiQXVKNCtWMCtnWHlEd1BHaUpPUXdoNG9NS00zOFJXZ2UzMHhqR1AwUnRVNEQ3Z2VxUFh5Ck5nVDR3cHphREU2UWk0S0t3endwZGV4MHovMUJQeWxtTlo3Um9GbDNJUTZkR2l2MTNLekFhK1BJbkdjK052N1EKZ0I5bDc0QTRBYWxWeTBmV3NITzNIcEI2RGJPRTUvbmRDOHJmTEhlMmE1SXdFR09pRGhoNFdrVjNmVVRvYkc3bwpOWEoydlhPZmJFekY2T3pPL1VTeEExSTBwUENkUTdKbUVQYytrTGw4WXlJczJKcFJNa0dybTF5TGpSV1hpVXFnCmVVY2NDck8yZVliUHdKbkJUckFrNjgwekVwTm5JV2QwTGlDWDIyaUFCSFY4OS9HTzlhM1crMWFaR3M0S0E5dXIKZ0VQem1ZSmVrblhzUlIxamdJTUNBd0VBQWFCSU1FWUdDU3FHU0liM0RRRUpEakU1TURjd0tBWURWUjBSQkNFdwpINElkWkdWMkxtRnlaMjlqWkM1bGQyMHpMbUZ6YzJWMGJXRnlheTV1WlhRd0N3WURWUjBQQkFRREFnV2dNQTBHCkNTcUdTSWIzRFFFQkN3VUFBNElCQVFBL3lPcVZNdWIrazZHUzNnek5PcmQrT3ZGQ213Q2RWZ3g4NCtGT0gwVlUKb1luTTVSUWcrZmJyOUI2eDVHRzA0WG9mTVl5eUJJenZVQzd5QUg3djN4UkhOZGJVeVNteWp0NEpGVWFQcDJlcwpuSXdNMm9pSklxbEM4UEFyeG1PYlpTMkZWWTZrcTVVVXYvVHV6VTZHaUd0aTBCSmcrTFhySTQ2U2xrTHM1OEdHCm5hUVJubGRpTW51ZlN3b1VFVG1CcW5OMm0zTC94MmxBRTlTQjZFN0ZVZk9FNlI3SXBVRVpyN3doTFFwRno0VjQKQ3RmNmJaYU1tMmJiOVh6NkRQK2VaSW9obTRUc08vQ3B3ZUhkVVFoM3NCQ0l5MmNlRnk0VFhkL1UrYUptL1dXWgorelRlUHNSNnA1KzJQQ2xGWGM4bnhXR284RmxvT0xNZjVJNmRRS1FHQlpmTgotLS0tLUVORCBDRVJUSUZJQ0FURSBSRVFVRVNULS0tLS0K
Status:
  Authorizations:
    Challenges:
      Token:        e8t60kb3mDeO5BWHAkYg_RI1dXH57z_ZBZZeLbyUa6U
      Type:         http-01
      URL:          https://acme-v02.api.letsencrypt.org/acme/chall-v3/278379837496/W_Tvfg
      Token:        e8t60kb3mDeO5BWHAkYg_RI1dXH57z_ZBZZeLbyUa6U
      Type:         dns-01
      URL:          https://acme-v02.api.letsencrypt.org/acme/chall-v3/278379837496/22QXhQ
      Token:        e8t60kb3mDeO5BWHAkYg_RI1dXH57z_ZBZZeLbyUa6U
      Type:         tls-alpn-01
      URL:          https://acme-v02.api.letsencrypt.org/acme/chall-v3/278379837496/wE0I2Q
    Identifier:     argocd.yourdomain.com
    Initial State:  pending
    URL:            https://acme-v02.api.letsencrypt.org/acme/authz-v3/278379837496
    Wildcard:       false
  Certificate:      LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUZCekNDQSsrZ0F3SUJBZ0lTQThJZ01hMWc0eXBuaU9Ud2ZVMUh0ZW1TTUEwR0NTcUdTSWIzRFFFQkN3VUEKTURJeEN6QUpCZ05WQkFZVEFsVlRNUll3RkFZRFZRUUtFdzFNWlhRbmN5QkZibU55ZVhCME1Rc3dDUVlEVlFRRApFd0pTTXpBZUZ3MHlNekV3TWpneU1ETTBNVEZhRncweU5EQXhNall5TURNME1UQmFNQ2d4SmpBa0JnTlZCQU1UCkhXUmxkaTVoY21kdlkyUXVaWGR0TXk1aGMzTmxkRzFoY21zdWJtVjBNSUlCSWpBTkJna3Foa2lHOXcwQkFRRUYKQUFPQ0FROEFNSUlCQ2dLQ0FRRUF1dEIyUGJMZXNXb2lyaVZzQzRuajVYVDZCZklQQThhSWs1RENIaWd3b3pmeApGYUI3ZlRHTVkvUkcxVGdQdUI2bzlmSTJCUGpDbk5vTVRwQ0xnb3JEUENsMTdIVFAvVUUvS1dZMW50R2dXWGNoCkRwMGFLL1hjck1CcjQ4aWNaejQyL3RDQUgyWHZnRGdCcVZYTFI5YXdjN2Nla0hvTnM0VG4rZDBMeXQ4c2Q3WnIKa2pBUVk2SU9HSGhhUlhkOVJPaHNidWcxY25hOWM1OXNUTVhvN003OVJMRURValNrOEoxRHNtWVE5ejZRdVh4agpJaXpZbWxFeVFhdWJYSXVORlplSlNxQjVSeHdLczdaNWhzL0FtY0ZPc0NUcnpUTVNrMmNoWjNRdUlKZmJhSUFFCmRYejM4WTcxcmRiN1Zwa2F6Z29EMjZ1QVEvT1pnbDZTZGV4RkhXT0Fnd0lEQVFBQm80SUNIekNDQWhzd0RnWUQKVlIwUEFRSC9CQVFEQWdXZ01CMEdBMVVkSlFRV01CUUdDQ3NHQVFVRkJ3TUJCZ2dyQmdFRkJRY0RBakFNQmdOVgpIUk1CQWY4RUFqQUFNQjBHQTFVZERnUVdCQlROV2xpay8remRSUmFqbGtEZGxtVCtSbVJTOWpBZkJnTlZIU01FCkdEQVdnQlFVTHJNWHQxaFd5NjVRQ1VEbUg2K2RpeFRDeGpCVkJnZ3JCZ0VGQlFjQkFRUkpNRWN3SVFZSUt3WUIKQlFVSE1BR0dGV2gwZEhBNkx5OXlNeTV2TG14bGJtcUdTSWIzRFFFQkN3VUFBNElCQVFBS2N3QnNsbTcvRGxMUXJ0Mk01MW9HclMrbzQ0Ky95UW9ERlZEQwo1V3hDdTIrYjlMUlB3a1NJQ0hYTTZ3ZWJGR0p1ZU43c0o3bzVYUFdpb1c1V2xIQVFVN0c3NUsvUW9zTXJBZFNXCjlNVWdOVFA1MkdFMjRIR050TGkxcW9KRmxjRHlxU01vNTlhaHkyY0kycUJETEtvYmt4L0ozdldyYVYwVDlWdUcKV0NMS1RWWGtjR2R0d2xmRlJqbEJ6NHBZZzFodG1mNVg2RFlPOEE0anF2MklsOURqWEE2VVNiVzFGelhTTHI5TwpoZThZNElXUzZ3WTdiQ2tqQ1dEY1JRSk1FaGc3NmZzTzN0eEUrRmlZcnVxOVJVV2hpRjFteXY0UTZXK0N5QkZDCkRmdnA3T09HQU42ZEVPTTQrcVI5c2Rqb1NZS0VCcHNyNkd0UEFRdzRkeTc1M2VjNQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg==
  Finalize URL:     https://acme-v02.api.letsencrypt.org/acme/finalize/819289237/218302940896
  State:            valid
  URL:              https://acme-v02.api.letsencrypt.org/acme/order/819289237/218302940896
Events:             <none>

In case of failure, here we can see that cert-manager has created two Challenge resources to verify we control specific domains, a requirements of the ACME order to obtain a signed certificate.

Step-6: Troubleshooting Challenges

In order to determine why an ACME Order is not being finished, we can debug using the Challenge resources that cert-manager has created.

kubectl get challenges -A
kubectl get challenges -n argocd

# output
NAME                                 STATE     DOMAIN            REASON                                     AGE
example-com-2745722290-4391602865-0  pending   example.com       Waiting for dns-01 challenge propagation   22s

Get more information about a specific Challenge:

 kubectl describe challenge example-com-2745722290-4391602865-0

# output
Status:
  Presented:   true
  Processing:  true
  Reason:      Waiting for dns-01 challenge propagation
  State:       pending
Events:
  Type    Reason     Age   From          Message
  ----    ------     ----  ----          -------
  Normal  Started    19s   cert-manager  Challenge scheduled for processing
  Normal  Presented  16s   cert-manager  Presented challenge using dns-01 challenge mechanism

Conclusion By systematically checking these resources and their statuses, you can identify and resolve issues with Let's Encrypt certificates managed by cert-manager. Refer to cert-manager's official troubleshooting documentation for additional insights.