Debugging with AI
An agent debugs as well as the evidence you give it. A staff engineer's job is to make evidence cheap to collect: shared scripts that collect logs, traces, and system state in a standard format, checked into the repo so every engineer and every agent uses the same tools.
Scripts worth having:
logs.sh: recent logs for a service, from wherever they livetrace.sh: a distributed trace by IDprofile.sh: CPU and memory profile of a processdebug.sh: one-shot system summary
Example debugging scripts
scripts/logs.sh - recent logs for a service:
#!/bin/bash
# Usage: ./scripts/logs.sh [service] [minutes]
SERVICE=${1:-"api"}
MINUTES=${2:-5}
echo "=== Last $MINUTES minutes of $SERVICE logs ==="
echo "=== Timestamp: $(date -u +%Y-%m-%dT%H:%M:%SZ) ==="
if docker ps --format '{{.Names}}' | grep -q "$SERVICE"; then
docker logs --since "${MINUTES}m" "$SERVICE" 2>&1 | tail -200
elif kubectl get pods -l app="$SERVICE" &>/dev/null; then
kubectl logs -l app="$SERVICE" --since="${MINUTES}m" --tail=200
elif [ -f "logs/$SERVICE.log" ]; then
tail -200 "logs/$SERVICE.log"
else
echo "No logs found for $SERVICE"
fiscripts/trace.sh - one request trace:
#!/bin/bash
# Usage: ./scripts/trace.sh <trace_id>
TRACE_ID=$1
[ -z "$TRACE_ID" ] && { echo "Usage: ./scripts/trace.sh <trace_id>"; exit 1; }
echo "=== Trace: $TRACE_ID ==="
# Jaeger query API; adapt the URL for Tempo or your vendor
curl -s "http://localhost:16686/api/traces/$TRACE_ID" \
| jq '.data[0].spans[] | {service: .process.serviceName, op: .operationName, duration: .duration}'scripts/debug.sh - system summary:
#!/bin/bash
# Usage: ./scripts/debug.sh
echo "=== System Diagnostics $(date) ==="
echo -e "\n=== Containers ==="
docker ps --format "table {{.Names}}\t{{.Status}}\t{{.Ports}}" 2>/dev/null || echo "Docker not running"
echo -e "\n=== Listening Ports ==="
lsof -i -P -n | grep LISTEN | awk '{print $1, $9}' | sort -u
echo -e "\n=== Disk ==="
df -h | grep -E '^/dev|Filesystem'
echo -e "\n=== Recent Errors ==="
grep -i "error\|exception\|fatal" logs/*.log 2>/dev/null | tail -50
echo -e "\n=== Environment ==="
echo "NODE_ENV: ${NODE_ENV:-not set}"
echo "DATABASE_URL: ${DATABASE_URL:+[REDACTED]}"
echo "API_KEY: ${API_KEY:+[REDACTED]}"For profiling, use the standard tool per runtime rather than a homegrown wrapper: node --cpu-prof, py-spy record, Go's pprof.
Using scripts with AI
Better still, let the agent run the scripts itself:
"Users see 500s on /checkout. Run ./scripts/logs.sh api 10 and ./scripts/debug.sh,
then tell me the likely cause and how to confirm it."Example debugging prompts:
Bug report:
"Checkout fails intermittently. ~10% of requests, worse at peak, no Sentry errors,
users see 'Something went wrong'. Build a debugging plan, cheapest checks first."
Regression:
"This error started after yesterday's deploy. Stack trace: [paste]. Diff: PR #412.
What's the likely cause?"
Performance:
"/api/products takes 3-5s. Trace shows a 2.8s query: [paste]. Suggest fixes and
how to verify each."
Stuck:
"Tried: restart (no change), rollback (still happens), DB connections (healthy).
What next? Give me exact commands."
Reproduce first:
"Write a test that reproduces this bug: [describe]. We fix it once it fails."Rules for incidents:
- Agents are read-only in production by default. Write actions (restarts, rollbacks, data fixes) go behind a human approval
- Plausible is not proven. Don't accept a root cause the agent can't reproduce or support with evidence
- Turn findings into checks. Every postmortem finding becomes a test, an alert, or a CLAUDE.md rule, so the same class of bug stays fixed