Provider Troubleshooting

Issues with LLM providers — auth failures, circuit breakers, and connectivity.

Check live reachability first

GET /gateway/pools reports live, per-provider reachability for every configured local and cloud provider — auth status, circuit-breaker state, and the models each one exposes:

curl http://localhost:4200/gateway/pools -H "Authorization: Bearer hzk_..." | jq

Each provider entry includes auth_status (Configured/Missing/NotRequired), reachable (circuit-breaker closed/open), and sovereignty_reason (why it's classified local vs. cloud). For a quick CLI summary without the raw JSON:

hoziron-cli providers list

Provider auth failure

Symptoms: "Provider authentication failed" or "401 Unauthorized"

Diagnosis:

# Test the provider key against the live daemon config
hoziron-cli config test-key anthropic

# Check if the env var is set
hoziron-cli config get providers.anthropic.api_key_env
echo $ANTHROPIC_API_KEY

Fix:

  1. Verify the API key is valid (not expired, not revoked at the provider's dashboard)
  2. Ensure the env var is available to the daemon process: 
    # If using systemd, add to the service:
Environment=ANTHROPIC_API_KEY=sk-ant-...
  3. Re-set the key: hoziron-cli config set-key anthropic — or store it in the encrypted credential vault instead: hoziron-cli vault set ANTHROPIC_API_KEY
  4. Restart if the env var was changed externally (config reload handles file changes, not env changes)

Provider keys are resolved lazily — a missing key won't error at startup, it fails on first use. hoziron-cli providers list / GET /gateway/pools will show auth_status: Missing before you ever send a request.

Circuit breaker tripped

Symptoms: Requests to a provider fail immediately. GET /gateway/pools shows reachable: false for that provider.

Fix:

The circuit breaker auto-recovers after a cooldown: after [health].recovery_cooldown_secs (default 60s), one probe request is sent (half-open). A successful probe closes the circuit and resets the failure count; a failure trips it straight back to open.

To tune:

  • [health].failure_threshold (default 5) — consecutive failures to trip
  • [health].recovery_cooldown_secs (default 60) — seconds before the next probe

If the provider is genuinely down, configure a fallback via complexity routing:

[routing]
simple_model = "groq/llama-3.1-8b-instant"
medium_model = "anthropic/claude-sonnet-4-20250514"
complex_model = "openai/gpt-4o"

Provider URL not reachable (Docker / air-gapped)

Symptoms: Connection timeout or "connection refused" to Ollama or vLLM.

Common causes:

  • Ollama bound to 127.0.0.1 but Hoziron is in a container
  • Wrong port or hostname

Fix:

  • Start Ollama with OLLAMA_HOST=0.0.0.0 ollama serve
  • In Docker, use --add-host=host.docker.internal:host-gateway and set: 
    [providers.ollama]
base_url = "http://host.docker.internal:11434"
  • Verify from inside the container: 
    wget -qO- http://host.docker.internal:11434/api/tags
  • Confirm the daemon sees it: GET /gateway/pools should list the provider under local with reachable: true.

Rate limiting (429)

If a provider returns 429, Hoziron retries with exponential backoff. If all retries fail and fallback models are configured, the next model in the chain is tried.

To reduce rate limiting:

  • Use complexity routing to spread load across providers
  • Reduce concurrent agent count
  • Check your provider's rate limit tier

Related: