Performance Troubleshooting

High memory, slow responses, and optimization strategies.

Check /health and /metrics first

GET /health includes latency_p95_ms — a rolling p95 of request latency since boot. It is null until at least one request has completed, or if metrics are disabled entirely:

hoziron-cli health --json
# {"status": "...", "uptime_secs": ..., "latency_p95_ms": 842, "agents": {...}, "providers": {...}}

If latency_p95_ms stays null after real traffic, metrics collection is disabled — enable it in config (see the observability guide).

For deeper investigation, GET /metrics exposes a Prometheus-format endpoint gated on metrics being enabled — it returns 404 {"error": {"category": "NotFound", "message": "Metrics not enabled"}} if not:

curl http://localhost:4200/metrics

High memory usage

Diagnosis:

hoziron-cli status --json
# Check agent count and memory subsystem

ps aux | grep hoziron

Root causes:

• Each agent holds conversation history in memory
• Long-lived agents accumulate context over time
• Many Running agents each hold base memory (~10 MB)

Fix:

• Reset an agent's session: POST /agents/{id}/session/reset
• Clear history: DELETE /agents/{id}/history
• Suspend idle agents: hoziron-cli agent suspend <id> (agents have two states — Suspended/Running — see agent troubleshooting)
• Schedule agents (activate on demand via cron) rather than leaving them Running perpetually

Slow responses

Diagnosis:

1. Check latency_p95_ms from /health — is the platform itself slow, or is it provider latency?
2. Cross-reference against GET /gateway/pools — a provider showing reachable: true but consistently slow responses points at the provider, not Hoziron.
3. Check timeout config:

   hoziron-cli config get surfaces.api.limits.request_timeout_secs
   # Default: 600 (10 minutes) — enforced per listener (Issue #682)
   

Fix:

• For faster responses, use a faster model (Groq with Llama is significantly faster than Claude/GPT-4)
• Configure complexity routing:

  [routing]
  simple_model = "groq/llama-3.1-8b-instant"
  complex_model = "anthropic/claude-sonnet-4-20250514"
  simple_threshold = 100
  complex_threshold = 500
  

• Check provider quotas — you may be rate-limited at the provider level (see provider troubleshooting)

Capacity guidelines

Per-agent overhead

• Memory: ~10 MB base + conversation context (proportional to history length)
• Storage: ~1 MB per 1000 messages
• CPU: Negligible when idle; bursts during LLM interactions

Scaling considerations

• Hoziron is single-process, single-node by design (simplicity over distribution)
• Scale vertically for more agents on one instance
• For multi-team isolation, run separate instances per team
• Per-provider circuit breakers protect against provider overload — see provider troubleshooting
• The kernel recovery coordinator retries up to 3 consecutive times (with cooldown) on subsystem failure — see recovery procedures

Storage breakdown

• data/hoziron.db — main state database (agents, sessions, config)
• data/memory/ — per-agent KV stores
• data/audit/ — audit trail with Merkle chain
• data/packages/ — installed catalog packages

If storage is growing fast, check audit trail retention ([audit].max_entries, default 100,000) and agent history accumulation.

Related:

• Common issues
• Observability guide
• Recovery procedures