Troubleshooting: "Connection Timed Out" Errors in Dashboard

Symptom Description

Users see a "Connection timed out" error when loading the Analytics Dashboard. The error appears after a ~30 second delay. Other pages load normally.

Error: Request to /api/v2/analytics/dashboard failed
Status: 504 Gateway Timeout
Message: "Connection timed out after 30000ms"

Possible Causes

1. Database query timeout (most likely) — The dashboard query exceeds the 30s timeout
2. API server resource exhaustion — Too many concurrent requests consuming all workers
3. Network connectivity — Intermittent connectivity between API server and database
4. Data volume spike — Sudden increase in data volume causing slower queries

Diagnostic Steps

Step 1: Check API Server Health

curl -s https://api.internal/health | jq '.status'
# Expected: "ok"
# If not "ok": Go to Solution B

Step 2: Check Slow Query Log

kubectl logs deploy/api-server --since=10m | grep "SLOW_QUERY"
# If slow queries found: Go to Solution A

Step 3: Check Database Connections

psql -c "SELECT count(*) as total, state FROM pg_stat_activity GROUP BY state;"
# If 'active' count is near max_connections: Go to Solution B

Solutions

Solution A: Optimize Slow Query

The dashboard query likely needs an index. Run:

CREATE INDEX CONCURRENTLY idx_events_dashboard
ON events (org_id, created_at DESC, event_type);

Verify with: EXPLAIN ANALYZE on the dashboard query — should show Index Scan instead of Seq Scan.

Solution B: Restart API Workers

If the server is resource-exhausted:

kubectl rollout restart deploy/api-server
kubectl rollout status deploy/api-server
# Wait for all pods to be Ready

Prevention

• Add a database query timeout of 10s (fail fast instead of hanging)
• Set up monitoring alert for queries exceeding 5s
• Schedule weekly ANALYZE on high-traffic tables