Observability

What you'll accomplish: Configure logging (including real HOZIRON_LOG_FORMAT=json structured output), scrape Prometheus metrics, and enable the new workflow_step/llm_call tracing spans with optional OpenTelemetry export.

Logging

Filtering

HOZIRON_LOG=info hoziron-server
HOZIRON_LOG="info,hoziron_core::health=debug,hoziron_api=warn" hoziron-server

HOZIRON_LOG accepts standard tracing-subscriber EnvFilter directives. If unset, it falls back to RUST_LOG, then to info.

Output goes to two sinks simultaneously: stderr, and a daily-rotating file at $HOZIRON_HOME/logs/server.log.

HOZIRON_LOG_FORMAT=json for structured logs

hoziron-server --help documents HOZIRON_LOG_FORMAT ("json or text, default text"), and docker-compose.yml passes it straight through as an environment variable. Setting HOZIRON_LOG_FORMAT=json (case-insensitive) switches both the stderr sink and the daily-rotating $HOZIRON_HOME/logs/server.log file sink to tracing_subscriber::fmt::layer().json() — structured JSON lines instead of the default human-readable text formatter. Any other value (or leaving it unset) keeps plain text.

If you're feeding logs into a collector that expects JSON (Fluentd, Loki, CloudWatch Logs Insights with a JSON parser), set HOZIRON_LOG_FORMAT=json and point the collector at a JSON parser.

Prometheus metrics

GET /metrics is served in Prometheus exposition format whenever the API surface is active — there's no separate config toggle to enable or disable it; it's installed unconditionally as part of booting the API surface, on the same port as the API (default 4200):

curl http://localhost:4200/metrics

A registry-only, MCP-only, or dashboard-only deployment (API surface not in --surfaces) has no /metrics endpoint at all — the recorder is only installed in the branch of main.rs that boots the API surface.

Scrape config

scrape_configs:
  - job_name: hoziron
    static_configs:
      - targets: ["hoziron:4200"]
    metrics_path: /metrics

Real metric names

Confirmed from crates/platform/hoziron-core/src/observability/metrics.rs:

Tracing spans: workflow_step and llm_call

Two structured tracing::info_span! spans were recently added and are compiled in by default (no feature flag) — they show up in the regular text logs as nested spans with recorded fields, whether or not OpenTelemetry export is enabled:

• workflow_step (crates/kernel/hoziron-kernel/src/workflow.rs) — one per workflow step execution. Fields: workflow_id, step_index, step_name, mode, iteration (loop steps only), correlation_id, and recorded on completion: duration_ms, input_tokens, output_tokens, success.
• llm_call (crates/kernel/hoziron-runtime/src/agent_loop.rs) — one per model invocation. Fields: provider, model, and recorded on completion: input_tokens, output_tokens, duration_ms.

llm_call spans nest inside their parent agent-loop iteration; workflow_step spans nest inside the workflow_run span (which lives on WorkflowEngine::execute_run_from, spanning the run's actual duration — a prior version of this span was entered and dropped before any step ran, so it never wrapped anything real; that's fixed). These are visible directly in the text logs at info level with no extra configuration — they don't require the telemetry feature or OTLP.

OpenTelemetry (optional, build-time feature, off by default)

OTLP span export is gated at compile time behind the telemetry Cargo feature — off unless the binary was explicitly built with it — and at runtime behind [telemetry].enabled in config.toml. The standard release build and the published Docker image do not include OTLP export at all — zero overhead, as intended. If you need OTLP export, ask Hoziron for a telemetry-enabled build; there is no way to toggle this on a standard binary.

[telemetry]
enabled = true
endpoint = "http://otel-collector:4317"   # falls back to OTEL_EXPORTER_OTLP_ENDPOINT env var, then http://localhost:4317
service_name = "hoziron"

When enabled, every tracing span (including workflow_step and llm_call above) gets bridged to an OTLP gRPC exporter via tracing-opentelemetry. If the exporter can't be constructed (e.g. bad endpoint), the server logs a warning and continues without it — it does not fail to boot.

Health endpoints

allowed_ips on a surface's listener ([surfaces.<name>].allowed_ips, real and enforced — see TLS and networking § IP allowlist) applies to /__server/health and /metrics like any other path on that listener. The one exemption the middleware makes by path is a surface's own /health route (e.g. the API surface's /health), which always bypasses regardless of the allowlist. If you restrict allowed_ips on the API surface's listener, make sure your Prometheus scraper's and load balancer's source IPs are included, or point health probes at the exempted /health path instead of /__server/health.

Environment variables summary

Next steps

• Docker Compose deployment — why the healthcheck uses --version instead of an HTTP probe
• Troubleshooting performance

Related:

• Docker Compose deployment
• Troubleshooting performance