PII Engine

What you'll accomplish: Understand how the PII Engine detects, classifies, tokenizes, and hydrates personally identifiable information to enforce data sovereignty before messages reach models or external tools.

Overview

The PII Engine is a runtime subsystem that intercepts messages flowing between agents and model providers. It ensures that sensitive data (national IDs, email addresses, financial numbers, etc.) never leaves the deployment boundary unless explicitly permitted by a trust policy.

The engine operates in two directions:

• Outbound (tokenization): Messages heading to models or tools are scanned for PII, which is replaced with opaque tokens.
• Inbound (hydration): Responses returning to trusted destinations can have tokens restored to their original values.

Architecture

Pipeline stages

Configuration

ADR-049 (Breaking Change): PII policy is now exclusively operator-owned via the carrier policy file. The per-agent trust_policy field and PUT /agents/{id}/trust-policy endpoint have been removed. Agents have no PII-egress input. All configuration below applies at the platform level.

PII behavior is controlled via carrier-pii-policy.toml — a single operator-owned, file-sourced policy reloaded on restart or file-watch. No runtime mutation API exists.

Carrier PII policy file

Location: $HOZIRON_HOME/carrier-pii-policy.toml

# ── LLM dimension ────────────────────────────────────────────────
# Default: cloud_permitted (PII tokenised before egress).
# Named types may not leave the box even tokenised.
[pii_policy.llm]
local_only = ["sa_id_number", "bank_account", "medical_record"]

# ── Tool dimension ───────────────────────────────────────────────
# Default for any (destination, PII-type) with no explicit ruling: TOKENISE.
[[pii_policy.tool_rules]]
destination = "claims-core-sor"
hydrate     = ["sa_id_number", "bank_account"]
reject      = ["medical_record"]

[[pii_policy.tool_rules]]
destination = "notify-sms"
hydrate     = ["za_phone"]

LLM dimension — binary egress

Tool dimension — per-destination rules

Actions are resolved per-(destination, PII-type) pair. When multiple PII types co-occur in a single call, each type is resolved independently: one type may hydrate while another tokenises in the same call.

Key principles

• Tokenisation is the unconditional substrate — any PII in any payload is tokenised before egress, always.
• No agent-supplied value can relax enforcement — the carrier policy file is the sole authority.
• Deny-by-default — no path to hydration exists without an explicit hydrate entry.
• Case-insensitive destination matching — Claims-Core-SoR matches claims-core-sor.

NER score-shaping table

Alongside the regex + checksum tier, the platform can run a probabilistic detector for entity classes no regular expression can express — principally person names and free-form addresses. That detector's raw confidence scores are shaped by a static, operator-declared table before a span ever reaches the pipeline: per-category confidence thresholds, per-category trust multipliers, and an outright drop list. Nothing in this table is learned or adaptive — it is a table you edit and reload, the same as the rest of carrier-pii-policy.toml.

This is how a false positive gets tuned out without retraining or a code change. For example, if the detector occasionally mistags a role noun like "assessor" as a person's name, raising the person_name threshold suppresses that specific class of error the next time the policy file is reloaded.

[pii_policy.ner_scoring]
# Per-category confidence threshold override. A category with no entry
# here keeps the detector's own default (0.5).
thresholds = { person_name = 0.8, physical_address = 0.6 }

# Per-category trust multiplier, applied to the raw confidence score before
# the threshold check above. Down-weights a category known to over-fire.
# Must be in (0.0, 1.0] — this table only ever down-weights, never boosts.
trust_multipliers = { physical_address = 0.9 }

# Categories dropped outright from the probabilistic tier's output on this
# deployment, regardless of confidence.
drop = []

Every category named in thresholds, trust_multipliers, or drop is checked against the same fail-closed category registry as the rest of the file: a category no detector on this deployment can emit — including person_name when no model is loaded — is rejected at load time, not silently ignored.

The category name is the config entry an auditor traces a score adjustment back to: every detected span's audit record carries its category and the confidence that gated its emission, so "why was this tokenised" always resolves to a single named row in this table.

Regions

The pattern registry covers five regions with validated patterns:

• South Africa — SA ID numbers (YYMMDD + Luhn), phone numbers
• United Kingdom — NHS numbers (Modulus-11), NI numbers, phone numbers
• Europe — IBAN (mod-97), French INSEE numbers, German Personalausweis (check digit)
• North America — US SSN (area exclusion), Canadian SIN (Luhn), credit cards (Luhn)
• Universal — Email addresses, phone numbers (E.164), credit cards

To limit detection to specific regions:

[pii]
regions = ["SouthAfrica", "Universal"]

Token format

Tokens follow a fixed format for reliable detection during hydration:

<<PII:a1b2c3d4e5f6789012345678abcdef01>>

• Prefix: <<PII:
• Body: 32 hex characters (UUID v4, hyphen-stripped)
• Suffix: >>

This format is designed to be unlikely to appear in natural text while being machine-parseable.

Token Vault

The vault is an in-process SQLite database that stores token-to-value mappings. Every agent gets its own vault, unconditionally — ADR-049 made PII detection a platform constant, not an agent-level opt-in: build_pii_pipeline (agent/mod.rs) always constructs a full detection pipeline (all regions, all categories) for every agent, with no toggle to disable it. There is no "trust policy enabled" state to check — the per-agent vault exists from the moment the agent is created, providing memory isolation between agents.

Location: $HOZIRON_HOME/data/pii_vaults/{agent_id}.db

Lifecycle:

1. Tokens are stored when PII is tokenized (outbound path)
2. Tokens are looked up when hydration is requested (inbound path)
3. Expired tokens are treated as not found during lookup — this prevents stale PII from being restored after the retention window
4. A lazy purge removes expired rows when the vault is opened
5. Vault files persist even when the trust policy is disabled (audit retention)
6. Vault files are removed when the agent is uninstalled

Retention: Fixed at 24 hours (DEFAULT_TOKEN_RETENTION_HOURS in pii/vault.rs). The per-agent policy struct (AgentAccessBoundary) holds only allowed_tools — there is no PII or retention field on it, and no [pii_policy] retention key in carrier-pii-policy.toml either. Retention is a platform constant, not an operator-configurable value. Expired tokens cannot be hydrated even if rows haven't been physically purged yet.

Concurrency: The vault uses WAL mode and a Mutex wrapper for safe concurrent access across async tasks.

Routing strategies

Hydration at the tool seam (deny-by-default)

Hydration uses deny-by-default semantics: a PII token is only restored to its real value if the carrier policy explicitly lists it in the hydrate array for that destination. No entry = tokenised (safe, non-leaking).

Example configuration:

[[pii_policy.tool_rules]]
destination = "claims-core-sor"
hydrate     = ["sa_id_number", "bank_account"]
reject      = ["medical_record"]

LLM routing (local_only)

When the routing gateway detects a local_only PII type in a payload, the request is forced to a local model. No cloud ResolvedModelTarget is ever produced. The PII is still tokenised — local_only is an additional constraint on top of tokenisation.

[pii_policy.llm]
local_only = ["sa_id_number", "bank_account", "medical_record"]

Best for: highly sensitive data classes where even tokenised round-trips to cloud are unacceptable.

Tokenize-and-forward (default)

PII is replaced with tokens before the message reaches the model. The model sees <<PII:...>> placeholders and can reason about structure without accessing the actual values. This is the unconditional substrate — it fires on every message regardless of configuration.

Best for: cloud-hosted models where you need model intelligence but cannot expose PII.

Audit events

The PII engine emits structured tracing events for compliance:

These events integrate with the audit trail and are captured by JSON log output for SIEM forwarding.

Performance

The engine targets sub-10ms processing for a 4096-byte message:

• Regex compilation happens once at pipeline construction (not per-message)
• Token regex is compiled once in the hydrator constructor
• Detection uses regex-lite (no backtracking, predictable performance)
• Vault uses SQLite WAL mode for non-blocking reads

Disabling PII detection

PII detection cannot be disabled at runtime. The carrier policy file controls which PII types are detected and how they are handled. To effectively disable enforcement for a specific destination, list all PII types in its hydrate array — values will be restored to real form (functionally equivalent to no tokenisation for that destination).

To disable the PII engine entirely (development/testing only), remove the [pii_policy] section from carrier-pii-policy.toml. Without a carrier policy, the platform falls back to detect+tokenise for all destinations (safe default).

Next steps

• Audit trail — how PII events are logged
• Authentication — securing API access to the platform
• RBAC — config:write (Admin-only) is required to edit carrier-pii-policy.toml-adjacent config; there is no API endpoint that mutates PII policy at all — it is file-sourced only

Related:

• Security hardening checklist
• Deployment observability