Observability: Logs, Events, and Metrics

Troubleshooting Kubernetes starts with reading the signals the platform and application already provide: status, events, logs, metrics, and resource descriptions.

Kubernetes Field Notes · Troubleshooting Level · Estimated reading time: 22 minutes

What this lesson covers

Troubleshooting Kubernetes requires reading signals from the cluster and the workload. The first signals to learn are status, events, logs, metrics, and resource descriptions.

• What Kubernetes events tell you
• How logs fit into troubleshooting
• What describe output shows
• Why metrics may require extra components
• How to build a basic troubleshooting picture

The short version

get:
  What exists and what state is it in?

describe:
  What details and events explain the state?

logs:
  What did the application or container print?

events:
  What did Kubernetes try to do?

metrics:
  What resources are being used?

Before you start

kubectl config current-context
kubectl get nodes
kubectl create namespace signals-lab

Step 1: Create a normal Deployment

kubectl create deployment signals-web --image=nginx:latest -n signals-lab
kubectl expose deployment signals-web --port=80 --target-port=80 --type=ClusterIP -n signals-lab
kubectl get all -n signals-lab

Step 2: Review status quickly

kubectl get deployments -n signals-lab
kubectl get replicasets -n signals-lab
kubectl get pods -n signals-lab
kubectl get services -n signals-lab
kubectl get endpoints -n signals-lab

Step 3: Describe the Deployment and Pod

kubectl describe deployment signals-web -n signals-lab
kubectl get pods -n signals-lab
kubectl describe pod <pod-name> -n signals-lab

Step 4: Generate application logs

kubectl port-forward -n signals-lab service/signals-web 8080:80

Open http://localhost:8080 a few times, then press Ctrl+C.

kubectl get pods -n signals-lab
kubectl logs <pod-name> -n signals-lab

Step 5: Follow logs live

kubectl logs -f <pod-name> -n signals-lab

Press Ctrl+C to stop following logs.

Step 6: Create an image pull failure

Create a file named:

bad-image-signals.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: bad-image-signals
  namespace: signals-lab
spec:
  replicas: 1
  selector:
    matchLabels:
      app: bad-image-signals
  template:
    metadata:
      labels:
        app: bad-image-signals
    spec:
      containers:
      - name: app
        image: nginx:not-a-real-tag-for-lab

kubectl apply -f bad-image-signals.yaml
kubectl get pods -n signals-lab
kubectl describe pod <bad-image-pod-name> -n signals-lab

Events should show image pull errors.

Step 7: View events directly

kubectl get events -n signals-lab --sort-by=.metadata.creationTimestamp

Step 8: Create a crash loop

Create a file named:

crashloop-signals.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: crashloop-signals
  namespace: signals-lab
spec:
  replicas: 1
  selector:
    matchLabels:
      app: crashloop-signals
  template:
    metadata:
      labels:
        app: crashloop-signals
    spec:
      containers:
      - name: app
        image: busybox:latest
        command: ["sh", "-c", "echo Starting; echo Failing now; exit 1"]

kubectl apply -f crashloop-signals.yaml
kubectl get pods -n signals-lab
kubectl logs <crashloop-pod-name> -n signals-lab
kubectl logs <crashloop-pod-name> -n signals-lab --previous
kubectl describe pod <crashloop-pod-name> -n signals-lab

--previous is useful when a container has already restarted and you need logs from the prior failed instance.

Step 9: Metrics note

Metrics commands may not work in a basic kind cluster unless metrics-server is installed:

kubectl top nodes
kubectl top pods -n signals-lab

In production, metrics usually come from a monitoring stack such as metrics-server, Prometheus, a cloud monitoring agent, or another telemetry pipeline.

Step 10: Build a basic troubleshooting picture

kubectl get pods -n signals-lab
kubectl describe pod <pod-name> -n signals-lab
kubectl logs <pod-name> -n signals-lab
kubectl logs <pod-name> -n signals-lab --previous
kubectl get events -n signals-lab --sort-by=.metadata.creationTimestamp
kubectl get deployment <deployment-name> -n signals-lab -o yaml

Common signals and what they suggest

• Pending: Scheduling, storage, or admission issue.
• ImagePullBackOff: Image name, tag, registry, or credentials issue.
• CrashLoopBackOff: Application starts and exits repeatedly.
• OOMKilled: Container exceeded memory limit.
• No endpoints: Service selector, Pod readiness, or label issue.
• Rollout stuck: New Pods are not becoming available.

Step 11: Clean up

kubectl delete namespace signals-lab

Check your understanding

1. What does kubectl get show?
2. What does describe add?
3. What do events usually explain?
4. What do logs come from?
5. When is --previous useful?
6. Why might kubectl top not work?
7. What status suggests image pull trouble?
8. What status suggests repeated process failure?
9. What does no endpoints suggest?
10. Name five commands in a basic troubleshooting flow.