Kubernetes Troubleshooting

Kubernetes Troubleshooting Playbook

Kubernetes troubleshooting becomes easier when you follow a repeatable flow instead of guessing. This playbook ties the previous lessons together.

Kubernetes Field Notes · Troubleshooting Level · Estimated reading time: 24 minutes

What this lesson covers

This lesson pulls the previous lessons into a practical troubleshooting playbook. Instead of memorizing random commands, you will follow a repeatable flow.

  • How to start with context and Namespace
  • How to troubleshoot Pods
  • How to troubleshoot Deployments
  • How to troubleshoot Services and Ingress
  • How to troubleshoot configuration, storage, and scheduling
  • How to avoid making things worse during an incident
Field note: The goal is not to run every command. The goal is to move from broad symptoms to specific evidence.

The universal first checks

kubectl config current-context
kubectl config view --minify --output 'jsonpath={..namespace}'
kubectl get namespaces
kubectl get pods -A
kubectl get events -A --sort-by=.metadata.creationTimestamp

Pod troubleshooting flow

kubectl get pods -n <namespace>
kubectl get pod <pod-name> -n <namespace> -o wide
kubectl describe pod <pod-name> -n <namespace>
kubectl logs <pod-name> -n <namespace>
kubectl logs <pod-name> -n <namespace> --previous
kubectl get events -n <namespace> --sort-by=.metadata.creationTimestamp

Deployment troubleshooting flow

kubectl get deployments -n <namespace>
kubectl describe deployment <deployment-name> -n <namespace>
kubectl rollout status deployment/<deployment-name> -n <namespace>
kubectl rollout history deployment/<deployment-name> -n <namespace>
kubectl get replicasets -n <namespace>
kubectl get pods -n <namespace>

Service troubleshooting flow

kubectl get services -n <namespace>
kubectl describe service <service-name> -n <namespace>
kubectl get endpoints <service-name> -n <namespace>
kubectl get pods --show-labels -n <namespace>
kubectl port-forward -n <namespace> service/<service-name> 8080:<service-port>

Ingress troubleshooting flow

kubectl get ingress -n <namespace>
kubectl describe ingress <ingress-name> -n <namespace>
kubectl get ingressclass
kubectl get services -n <namespace>
kubectl get endpoints <service-name> -n <namespace>
kubectl get pods -n ingress-nginx
kubectl logs -n ingress-nginx <controller-pod-name>

ConfigMap and Secret troubleshooting flow

kubectl get configmaps -n <namespace>
kubectl get secrets -n <namespace>
kubectl describe pod <pod-name> -n <namespace>
kubectl get pod <pod-name> -n <namespace> -o yaml
kubectl get events -n <namespace> --sort-by=.metadata.creationTimestamp

Storage troubleshooting flow

kubectl get pvc -n <namespace>
kubectl describe pvc <pvc-name> -n <namespace>
kubectl get pv
kubectl get storageclass
kubectl describe pod <pod-name> -n <namespace>

Scheduling troubleshooting flow

kubectl get pod <pod-name> -n <namespace>
kubectl describe pod <pod-name> -n <namespace>
kubectl get nodes --show-labels
kubectl describe node <node-name>
kubectl get events -n <namespace> --sort-by=.metadata.creationTimestamp

Probe troubleshooting flow

kubectl get pods -n <namespace>
kubectl describe pod <pod-name> -n <namespace>
kubectl logs <pod-name> -n <namespace>
kubectl get endpoints <service-name> -n <namespace>

Decision table

  • Pending: Check scheduling, resources, PVCs, node selectors, taints, and events.
  • ImagePullBackOff: Check image name, tag, registry access, and pull secrets.
  • CrashLoopBackOff: Check logs, previous logs, command, args, config, and dependencies.
  • Running but not Ready: Check readiness probes and endpoints.
  • Service not working: Check selector, labels, endpoints, and targetPort.
  • Ingress not working: Check controller, ingress class, host/path, Service, and endpoints.
  • OOMKilled: Check memory limits and application memory behavior.
  • Rollout stuck: Check new ReplicaSet Pods, readiness, image pulls, and rollout status.

Safe incident habits

  • Confirm cluster context before changing anything.
  • Confirm Namespace before changing anything.
  • Capture current state before deleting or restarting.
  • Prefer reversible changes.
  • Do not hide evidence by deleting failing Pods too early.
  • Use rollout history before and after deployment changes.
  • Document what you changed and when.

Practice lab: create and diagnose a bad rollout

kubectl create namespace troubleshoot-lab
kubectl create deployment troubleshoot-web --image=nginx:latest -n troubleshoot-lab
kubectl expose deployment troubleshoot-web --port=80 --target-port=80 --type=ClusterIP -n troubleshoot-lab
kubectl set image deployment/troubleshoot-web nginx=nginx:badtag-lab -n troubleshoot-lab
kubectl rollout status deployment/troubleshoot-web -n troubleshoot-lab

Diagnose it: 

kubectl get pods -n troubleshoot-lab
kubectl describe deployment troubleshoot-web -n troubleshoot-lab
kubectl describe pod <bad-pod-name> -n troubleshoot-lab
kubectl get events -n troubleshoot-lab --sort-by=.metadata.creationTimestamp

Recover: 

kubectl rollout undo deployment/troubleshoot-web -n troubleshoot-lab
kubectl rollout status deployment/troubleshoot-web -n troubleshoot-lab
kubectl get pods -n troubleshoot-lab

Clean up

kubectl delete namespace troubleshoot-lab
Bottom line: Troubleshooting Kubernetes is a repeatable evidence workflow. Start with context and Namespace, inspect resource state, follow ownership, read events, read logs, and make the smallest safe change.

Check your understanding

  1. What should you check before changing a cluster?
  2. What should you check before changing namespace-scoped resources?
  3. What command shows rollout status?
  4. What command shows previous container logs?
  5. What usually explains Pending Pods?
  6. What usually explains Service failures?
  7. What usually explains Ingress failures?
  8. Why should you avoid deleting failing Pods too quickly?
  9. What is the recovery command for a bad Deployment rollout?
  10. What are five commands in your standard first-response flow?
Answer key
  1. Current context.
  2. Namespace.
  3. kubectl rollout status deployment/<name>.
  4. kubectl logs <pod> --previous.
  5. Scheduling, resources, storage, taints, selectors, or admission issues.
  6. Selectors, labels, endpoints, ports, or Pod readiness.
  7. Controller, class, hostname, path, Service, or endpoints.
  8. They may contain evidence in status, events, and logs.
  9. kubectl rollout undo deployment/<name>.
  10. Examples: current-context, get pods, describe pod, logs, get events, rollout status.
← Previous: Observability Signals Series index Next: Production Readiness →