Kubernetes Troubleshooting

Troubleshooting Wazuh on Kubernetes

Learn how to troubleshoot Wazuh like a Kubernetes operator: start with pods, PVCs, StorageClasses, services, events, memory limits, and container restart reasons before blaming Wazuh itself.

Kubernetes SOC Field Lab · Practical Security Operations

Domain alignment

Platform Operations SIEM Operations Kubernetes Security

What this lesson covers

Why a running dashboard does not mean Wazuh is fully healthy
How StatefulSets depend on PVCs
How to diagnose Pending pods and Pending PVCs
How to read namespace events
How to fix local-path StorageClass binding issues
How to diagnose CrashLoopBackOff and OOMKilled manager pods
How to increase Wazuh manager memory requests and limits in a local lab
Why LoadBalancer services may stay pending in a local lab
PowerShell alternatives for Linux commands such as grep

Field note: Wazuh on Kubernetes is a Kubernetes troubleshooting lab as much as it is a SOC lab. If storage, scheduling, memory, services, or endpoints are broken, Wazuh will not behave correctly.

The Wazuh health model

Do not treat the dashboard as the only health indicator. The dashboard can be Running while the indexer, manager master, manager worker, agents, or alert pipeline are still unhealthy.

Dashboard Running
  does not automatically mean
Indexer Ready
  does not automatically mean
Manager Ready
  does not automatically mean
Agents enrolled
  does not automatically mean
Telemetry is complete

Step 1: Check Wazuh pods

kubectl get pods -n wazuh -o wide

Pay attention to READY, STATUS, RESTARTS, and whether a pod has an assigned node.

What to notice

Pending: the pod has not scheduled yet. Check PVCs, resources, taints, and events.
CrashLoopBackOff: the container starts and then fails repeatedly. Check describe pod and previous logs.
Running but high restarts: the pod may be temporarily healthy but unstable.
0/1 Ready: the container is not passing readiness checks.

Step 2: Check StatefulSets and Deployments

kubectl get statefulset -n wazuh
kubectl get deployment -n wazuh

Wazuh indexer and manager components are stateful. They need persistent storage. The dashboard is a Deployment and can often start even when stateful components are blocked.

Step 3: Check PVCs before application logs

kubectl get pvc -n wazuh

If the PVCs are Pending, the Wazuh StatefulSet pods cannot schedule. Do not troubleshoot dashboard login, Wazuh rules, or SOC alerts yet. Fix storage first.

Step 4: Read namespace events

kubectl get events -n wazuh --sort-by=.lastTimestamp

Events usually tell you the real reason a pod or volume is stuck. Look for scheduling failures, provisioning failures, mount failures, image pull failures, readiness probe failures, or back-off messages.

Storage issue: no node was specified

If you see messages like these, the issue is storage binding:

failed to provision volume with StorageClass "wazuh-storage": configuration error, no node was specified

pod has unbound immediate PersistentVolumeClaims

This commonly happens with local-path style storage when the StorageClass does not wait for a pod to be scheduled before provisioning the volume.

Check the StorageClass

kubectl get storageclass wazuh-storage -o yaml

For a local lab using the rancher.io/local-path provisioner, the StorageClass should normally include:

volumeBindingMode: WaitForFirstConsumer

Fix the local Wazuh storage class file

Open the local Wazuh storage class manifest:

notepad .\envs\local-env\storage-class.yaml

A local-path version should look similar to this:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: wazuh-storage
provisioner: rancher.io/local-path
reclaimPolicy: Delete
volumeBindingMode: WaitForFirstConsumer

Clean up failed pending PVCs

If this is a fresh lab and the PVCs never bound, delete the failed pending PVCs and the broken StorageClass, then reapply the fixed manifest.

kubectl delete pvc -n wazuh wazuh-indexer-wazuh-indexer-0
kubectl delete pvc -n wazuh wazuh-manager-master-wazuh-manager-master-0
kubectl delete pvc -n wazuh wazuh-manager-worker-wazuh-manager-worker-0

kubectl delete storageclass wazuh-storage

kubectl apply -k envs/local-env/

Watch PVCs bind

kubectl get pvc -n wazuh -w

You want the PVCs to move from Pending to Bound. Press Ctrl+C when they are bound.

Watch pods start

kubectl get pods -n wazuh -w

Expected progression:

Pending
ContainerCreating
Running
Ready

CrashLoopBackOff issue: manager pods are OOMKilled

If the Wazuh manager master or worker repeatedly restarts, check the pod details before trying to deploy agents.

kubectl describe pod -n wazuh wazuh-manager-master-0
kubectl describe pod -n wazuh wazuh-manager-worker-0

If the Last State section shows Reason: OOMKilled and Exit Code: 137, the container exceeded its memory limit and Kubernetes killed it.

Last State:     Terminated
  Reason:       OOMKilled
  Exit Code:    137

Limits:
  memory: 512Mi
Requests:
  memory: 512Mi

Do not continue to agent deployment yet: Wazuh agents depend on a healthy Wazuh manager. If the manager master or worker is in CrashLoopBackOff or has a rising restart count, fix the manager first.

Check whether the node has enough memory

kubectl describe node stealth-k8s-lab-control-plane

Look for Capacity, Allocatable, and Allocated resources. If the node has enough free memory, increase the Wazuh manager memory requests and limits.

Increase manager memory in the running lab

Use kubectl set resources. It is more reliable in PowerShell than a long JSON patch command.

kubectl set resources statefulset wazuh-manager-master -n wazuh -c wazuh-manager --requests=cpu=400m,memory=2Gi --limits=cpu=400m,memory=2Gi

kubectl set resources statefulset wazuh-manager-worker -n wazuh -c wazuh-manager --requests=cpu=400m,memory=2Gi --limits=cpu=400m,memory=2Gi

Restart the manager StatefulSets

kubectl rollout restart statefulset wazuh-manager-master -n wazuh
kubectl rollout restart statefulset wazuh-manager-worker -n wazuh

Watch the pods stabilize

kubectl get pods -n wazuh -w

You want both manager pods to become 1/1 Running. After a few minutes, run:

kubectl get pods -n wazuh
kubectl describe pod -n wazuh wazuh-manager-master-0
kubectl describe pod -n wazuh wazuh-manager-worker-0

Confirm the memory values changed to 2Gi and that new OOMKilled events are not appearing.

Make the memory fix permanent in the Wazuh manifests

The live fix updates Kubernetes, but it does not update the source manifests under C:\k8s\wazuh-kubernetes. Find the files that still reference 512Mi:

cd C:\k8s\wazuh-kubernetes
Get-ChildItem -Recurse -File | Select-String "512Mi"

Update the manager master and worker StatefulSet resource blocks from:

resources:
  limits:
    cpu: 400m
    memory: 512Mi
  requests:
    cpu: 400m
    memory: 512Mi

to:

resources:
  limits:
    cpu: 400m
    memory: 2Gi
  requests:
    cpu: 400m
    memory: 2Gi

Then reapply:

kubectl apply -k envs/local-env/

Operator note: Restart counts may not immediately reset to zero, but they should stop increasing after the memory limit is corrected. The important evidence is stable Running pods and no new OOMKilled events.

LoadBalancer EXTERNAL-IP pending in a local lab

In a local Kubernetes lab, services of type LoadBalancer may show EXTERNAL-IP <pending>. That is usually not the current problem unless your lab depends on an external load balancer controller.

kubectl get svc -n wazuh

For this lab, dashboard access uses port-forwarding, so a pending external IP does not block dashboard access.

PowerShell note: grep is not available by default

Some Kubernetes examples use Linux tools such as grep. In PowerShell, use Select-String instead.

kubectl get namespaces | Select-String wazuh

Or keep it simple:

kubectl get namespaces

Troubleshooting checklist

1. kubectl get pods -n wazuh -o wide
2. kubectl get statefulset -n wazuh
3. kubectl get deployment -n wazuh
4. kubectl get pvc -n wazuh
5. kubectl get storageclass wazuh-storage -o yaml
6. kubectl get events -n wazuh --sort-by=.lastTimestamp
7. kubectl describe pod -n wazuh <pod-name>
8. kubectl describe pvc -n wazuh <pvc-name>
9. kubectl logs -n wazuh <pod-name>
10. kubectl logs -n wazuh <pod-name> --previous --all-containers --tail=200
11. kubectl describe node stealth-k8s-lab-control-plane

Bottom line: Kubernetes events, PVC state, pod restart reasons, and resource limits usually tell you why Wazuh is not stable. Use platform evidence before guessing at the application.

Check your understanding

Why can the dashboard be Running while Wazuh is not fully healthy?
Why do Wazuh StatefulSets depend on PVCs?
What does a Pending PVC usually block?
What command shows namespace events?
What does no node was specified suggest in a local-path storage issue?
Why use WaitForFirstConsumer?
What does OOMKilled mean?
What does Exit Code 137 usually indicate in this context?
Why is kubectl set resources preferred here over a long JSON patch in PowerShell?
Why must manager pods be stable before deploying agents?
Why is EXTERNAL-IP <pending> not always a problem in a local lab?
What is the PowerShell alternative to grep?

Answer key

The dashboard is a separate Deployment and can start even if stateful manager/indexer pods are blocked.
They need persistent storage for stateful data.
It blocks the StatefulSet pod from scheduling.
kubectl get events -n wazuh --sort-by=.lastTimestamp.
The provisioner needed a selected node before creating local-path storage.
It waits to provision storage until Kubernetes knows which node will consume it.
The container exceeded its memory limit and was killed by the runtime.
The process was killed, commonly because of memory pressure inside the container limit.
It avoids fragile JSON quoting and path issues in PowerShell.
Agents need a healthy manager for enrollment, communication, and telemetry processing.
Local clusters often lack a load balancer controller, and this lab uses port-forwarding.
Select-String.