nanochat/docs/chaos-runbook.md

samosaChaat — Chaos Testing Runbook

This runbook covers failure scenarios for live defense. Each scenario includes how to simulate it, how to detect it via Grafana/Loki, and recovery steps.

Prerequisites

• kubectl configured for the target cluster
• Grafana accessible at https://grafana.samosachaat.art
• Loki datasource configured in Grafana

---

Scenario 1: Pod Crash / Kill

Simulate:

kubectl delete pod -l app.kubernetes.io/name=chat-api -n samosachaat-prod

Detect (Grafana):

• Dashboard: Application Performance → look for gap in request rate
• Alert: container restart count spike
• Panel query: kube_pod_container_status_restarts_total{namespace="samosachaat-prod"}

Detect (Loki):

{namespace="samosachaat-prod"} | json | level="error"
{namespace="samosachaat-prod",app="chat-api"} | json | message=~".*startup.*|.*shutdown.*"

Recovery:

• Kubernetes auto-restarts the pod (restartPolicy: Always)
• HPA scales up if CPU threshold exceeded during recovery
• PDB ensures other pods kept running during the kill
• No manual action needed unless crash-looping (check logs for root cause)

Verify recovered:

kubectl get pods -n samosachaat-prod -l app.kubernetes.io/name=chat-api
curl -s https://samosachaat.art/api/health | jq .

---

Scenario 2: Node Failure

Simulate:

# Get a node instance ID
INSTANCE_ID=$(kubectl get nodes -o jsonpath='{.items[0].spec.providerID}' | cut -d/ -f5)
aws ec2 terminate-instances --instance-ids $INSTANCE_ID

Detect (Grafana):

• Dashboard: Node Health → node disappears from CPU/Memory panels
• Alert: HighCPU or HighMemory may fire on remaining nodes as pods redistribute
• Panel query: kube_node_status_condition{condition="Ready",status="true"}

Detect (Loki):

{namespace="kube-system"} | json | message=~".*NotReady.*|.*node.*removed.*"

Recovery:

• EKS auto-scaling group launches a replacement node (2-5 minutes)
• Pods on the failed node are rescheduled to healthy nodes
• PDBs prevent more than 1 pod per service from being unavailable
• No manual action needed

Verify recovered:

kubectl get nodes    # New node should appear with STATUS Ready
kubectl get pods -n samosachaat-prod -o wide   # All pods Running

---

Scenario 3: Database Connection Pool Exhaustion

Simulate:

# Run a load test that exceeds the connection pool limit
kubectl run loadtest --image=busybox --restart=Never -n samosachaat-prod -- \
  sh -c 'for i in $(seq 1 200); do wget -q -O- http://chat-api:8002/api/health & done; wait'

Detect (Grafana):

• Dashboard: Application Performance → spike in p99 latency, increase in 5xx errors
• Alert: High5xxRate fires

Detect (Loki):

{app=~"auth|chat-api"} | json | message=~".*connection.*pool.*|.*timeout.*|.*asyncpg.*|.*QueuePool.*overflow.*"

Recovery:

1. Identify which service is affected from Loki logs
2. Check current connection count: kubectl exec deploy/chat-api -n samosachaat-prod -- python -c "..."
3. Restart affected pods: kubectl rollout restart deploy/chat-api -n samosachaat-prod
4. If persistent: increase pool size in service config (SQLALCHEMY_POOL_SIZE env var) and redeploy
5. Check RDS max_connections: aws rds describe-db-parameters --db-parameter-group-name default.postgres15

---

Scenario 4: Inference Service OOM

Simulate:

# Set a low memory limit and load a large model
kubectl set resources deploy/inference -n samosachaat-prod --limits=memory=512Mi
# Or trigger by sending many concurrent requests

Detect (Grafana):

• Dashboard: Inference Service → memory spike, then sudden drop (OOM kill)
• Dashboard: Node Health → memory spike on the node hosting inference
• Alert: HighMemory fires

Detect (Loki):

{app="inference"} | json | message=~".*OOMKilled.*|.*memory.*|.*killed.*"
# Also check events:
# kubectl get events -n samosachaat-prod --sort-by='.lastTimestamp' | grep -i oom

Recovery:

1. Pod auto-restarts (but may crash-loop if model is too large for limit)
2. Check what model is loaded: curl http://inference:8003/stats
3. If model too large: swap to smaller model via POST /models/swap
4. If limit too low: increase memory limit in values-prod.yaml and helm upgrade
5. Restore original limits: kubectl set resources deploy/inference -n samosachaat-prod --limits=memory=8Gi

---

Scenario 5: High Latency / Degraded Performance

Simulate:

# Flood inference with concurrent requests
kubectl run loadtest --image=curlimages/curl --restart=Never -n samosachaat-prod -- \
  sh -c 'for i in $(seq 1 50); do curl -s -X POST http://chat-api:8002/api/conversations/test/messages -H "Content-Type: application/json" -d "{\"content\":\"tell me a story\"}" & done; wait'

Detect (Grafana):

• Dashboard: Application Performance → p99 latency > 5s
• Dashboard: Inference Service → worker pool utilization at 100%, queue depth growing
• Alert: HighP99Latency fires

Detect (Loki):

{app="chat-api"} | json | inference_time_ms > 5000
{app="inference"} | json | message=~".*queue.*full.*|.*timeout.*|.*worker.*busy.*"

Recovery:

1. Check inference worker pool: curl http://inference:8003/stats
2. If all workers busy: HPA should scale inference pods (check HPA status)
3. Manual scale: kubectl scale deploy/inference -n samosachaat-prod --replicas=5
4. If single-pod bottleneck: check if model is too large for CPU inference, consider GPU nodes
5. Verify recovery: watch latency dashboard return to normal

---

Scenario 6: SSL Certificate Issues

Detect:

• Users report "connection not secure" errors
• curl -vI https://samosachaat.art 2>&1 | grep -i "expire\|ssl\|certificate"

Recovery:

• ACM certificates auto-renew — check ACM console for renewal status
• If DNS validation failed: check Route53 CNAME records match ACM requirements
• terraform apply to reconcile if records drifted

---

Quick Reference: Diagnostic Loki Queries

# All errors across all services
{namespace="samosachaat-prod"} | json | level="error" | line_format "{{.service}}: {{.message}}"

# Trace a request across services
{namespace="samosachaat-prod"} | json | trace_id="<TRACE_ID>"

# Auth failures
{app="auth"} | json | level="error" | message=~".*oauth.*|.*jwt.*|.*unauthorized.*"

# Inference issues
{app="inference"} | json | message=~".*error.*|.*timeout.*|.*OOM.*|.*worker.*"

# Slow database queries
{app=~"auth|chat-api"} | json | message=~".*slow.*query.*|.*timeout.*"

# Recent pod restarts
{namespace="samosachaat-prod"} | json | message=~".*started.*|.*shutdown.*|.*ready.*"