Kubernetes Pod troubleshooting
Introduction
There are not hard and fast rules for troubleshooting AKS pods, these are some basic general guidelines that may help for your troubleshooting.
Troubleshooting Kubernetes Pods can be a complex process, but there are several steps you can take to identify and resolve issues. Here are some general steps to follow when troubleshooting Kubernetes Pods:
login to Azure
Verify that you are logged into the right Azure subscription before start anything in visual studio code
Connect to Cluster
Use the following command to connect to your AKS cluster.
# Azure Kubernetes Service Cluster User Role
az aks get-credentials -g "rg-aks-dev" -n "aks-cluster1-dev"
# Azure Kubernetes Service Cluster Admin Role
az aks get-credentials -g "rg-aks-dev" -n "aks-cluster1-dev" --admin
# get nodes
kubectl get no
kubectl get namespace -A
Verify the pod status
This is the first thing we want to check, verify the pod status by running following commands.
If the Pod is in a "Pending" state, it may indicate a problem with the node or resource allocation. If the Pod is in a "Running" state, but the application is not working correctly, it may indicate an issue with the container or application configuration.
Describe the pod
To describe a pod in Kubernetes, you can use the kubectl describe command.
This can help you identify any issues with the container, such as a crash or resource allocation problems.
this command will also allow us to view the configuration for the Pod, including any environment variables or volume mounts. This can help you identify any issues with the Pod configuration that may be causing problems.
Check the logs
Use this command to view the logs generated by the pod and its containers. This can give you valuable information about what is causing the issue with the pod.
kubectl logs pod/aspnet-api-6b7d8fbf9f-mqtnd -n sample
# since 5 mins
kubectl logs --since=5m pod/aspnet-api-6b7d8fbf9f-mqtnd -n sample
This can help you identify any error messages or issues with the application.
Debug the container
For further debugging, let's dig into the container itself by running following commands, this will allows you to execute a command in a container in a pod.
kubectl exec -n sample -it aspnet-api-6b7d8fbf9f-mqtnd -- bash
# or
kubectl exec -n sample -it aspnet-api-6b7d8fbf9f-mqtnd -- /bin/sh
# or
kubectl exec -n sample -it aspnet-api-6b7d8fbf9f-mqtnd -- sh
if you want to check the deployed files run ls command inside the container.
root@aspnet-api-6b7d8fbf9f-mqtnd:/app# ls
AspNetApi AspNetApi.runtimeconfig.json Newtonsoft.Json.dll appsettings.Development.json
AspNetApi.deps.json Microsoft.AspNetCore.Mvc.Versioning.ApiExplorer.dll Swashbuckle.AspNetCore.Swagger.dll appsettings.json
AspNetApi.dll Microsoft.AspNetCore.Mvc.Versioning.dll Swashbuckle.AspNetCore.SwaggerGen.dll manifest
AspNetApi.pdb Microsoft.OpenApi.dll Swashbuckle.AspNetCore.SwaggerUI.dll web.config
kubectl exec command will also allow us to check network connectivity, use the kubectl exec command to connect to the container and test network connectivity to other services or resources.
Exit from the container
run the exit command to exit from running container.
install curl inside container
Use following apt commands to install tools as per the need while debugging.
output
Get:1 http://deb.debian.org/debian bullseye InRelease [116 kB]
Get:2 http://deb.debian.org/debian-security bullseye-security InRelease [48.4 kB]
Get:3 http://deb.debian.org/debian bullseye-updates InRelease [44.1 kB]
Get:4 http://deb.debian.org/debian bullseye/main amd64 Packages [8183 kB]
Get:5 http://deb.debian.org/debian-security bullseye-security/main amd64 Packages [229 kB]
Get:6 http://deb.debian.org/debian bullseye-updates/main amd64 Packages [14.6 kB]
Fetched 8635 kB in 1s (5772 kB/s)
Reading package lists... Done
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following additional packages will be installed:
.
.
.
curl internal pod URL
You can use the curl command inside the container to check the network connectivity of the pods installed on the same AKS cluster.
curl http://aspnet-api.namespace.svc.cluster.local:80/aspnetapi
curl http://aspnet-api.sample.svc.cluster.local:80/aspnetapi
Restart the pod:
To restart a pod in Kubernetes, you can use the kubectl delete pod command followed by the name of the pod you want to restart. When you delete a pod, Kubernetes will automatically create a new one to replace it.
Alternatively, you can use a deployment to manage the desired number of replicas of a pod, and perform a rolling update of the deployment to restart pods one by one. This approach avoids downtime and ensures high availability of your application.
It's also important to keep in mind that the troubleshooting steps may vary depending on the specific problem with the pod.
Print environment variables in pod
To print environment variables in a pod within a Kubernetes cluster, you can use following command.
kubectl exec aspnet-api-6b7d8fbf9f-mqtnd -n sample -- printenv
kubectl exec aspnet-api-6b7d8fbf9f-mqtnd -n sample -- env