# AEGIS — Data Flow Diagram

- **Document ID:** ARCH-10
- **Phase:** B.2 — Data, Evidence, Supply Chain
- **Status:** Draft for review (post two-reviewer discipline)
- **Version:** 0.1
- **Date:** 2026-07-10
- **Owner:** Chief Security Architect
- **Depends on:** ARCH-05, ARCH-06, ARCH-07, ARCH-08, ARCH-09
- **Consumed by:** ARCH-11 through ARCH-16

---

## 1. Purpose

Enumerate the material data flows through AEGIS, annotate each hop with its boundary label, capability, classification handling, and cryptographic-integrity contract, and expose the choke points where the Multi-Stage Decision Pipeline (ARCH-11) is applied.

A flow is presented as an ordered sequence of hops; each hop is annotated `[boundary | capability | classification-op | signature]`.

## 2. Flow F-1 — Ingest → Persist Raw

**Purpose.** Bring an external event across the perimeter, persist the raw copy immutably, and hand off to normalization.

| # | Hop | Annotation |
|---|---|---|
| 1 | Log source (Z7) → Connector Engine (Z2) | B-01 · source mTLS + rate limit · classification-tag (default C, source-configurable) · TLS 1.3 |
| 2 | Connector Engine → Kafka `ingest.raw.{tenant}.{source}` (Z3) | B-22 · `Publish-Ingest` · classification-carry · per-record integrity hash |
| 3 | Ack to log source | after persist to Kafka; never before |
| 4 | Storage Engine consumer → ClickHouse raw-table (Z3) | B-21 · `Write-Raw` (engine-scoped) · classification-carry · row includes source integrity hash + tenant DEK-encrypted payload |

**Sensitive-operation checklist:** Authorized ✓ (per-source capability), Logged ✓ (source connect, record accept, persist), Auditable ✓ (Layer-A per boundary), Traceable ✓ (`trace_id` propagated), Recoverable ✓ (Kafka replay window + immutable raw store).

## 3. Flow F-2 — Normalization

**Purpose.** Produce an additive OCSF-normalized view of every raw event, preserving raw verbatim.

| # | Hop | Annotation |
|---|---|---|
| 1 | Kafka `ingest.raw.*` → Normalization Engine | B-30x (broker) · `Read-Raw` · classification-carry |
| 2 | Normalization Engine → PII/secret scrubber sub-module | in-process · redaction manifest generated + signed |
| 3 | Normalization Engine → Kafka `events.normalized.{tenant}` | B-30x · `Publish-Normalized` · classification ≥ raw · normalized-record signed by Normalization Engine SVID |
| 4 | Storage Engine consumer → ClickHouse normalized-table | B-21 · `Write-Normalized` · classification enforcement · `raw_ref + integrity_hash` recorded (immutable event sourcing) |
| 5 | Unmapped events → Kafka `events.quarantine.{tenant}` | B-30x · `Publish-Quarantine` · isolated for later replay |

## 4. Flow F-3 — Correlation and Findings

| # | Hop | Annotation |
|---|---|---|
| 1 | Kafka `events.normalized.*` → Detection Engine | B-30x · `Read-Normalized` |
| 2 | Rule fires → Finding candidate emitted | in-process |
| 3 | Detection Engine → Kernel `Mediate(Publish-Finding)` | B-10 · policy consulted · Layer-A audit |
| 4 | Kernel → NATS `findings.{tenant}` | B-23 · signed finding record |
| 5 | Storage Engine consumer → PostgreSQL findings-table + ClickHouse evidence-link table | B-20 / B-21 · `Write-Finding` · tenant-scoped |

## 5. Flow F-4 — AI Recommendation (Multi-Stage Decision Pipeline)

This is the most consequential flow and where the elevation mandates (#3, #4, #5, #6, #9) apply structurally.

| # | Stage | Owner engine | Hop |
|---|---|---|---|
| 1 | **Trigger** | Case Engine or Detection Engine | Emits *RecommendationRequest* to NATS `ai.request.{tenant}` (`Request-AI-Recommendation`, classification per case) |
| 2 | **Evidence** | Evidence Engine (E-15) | Materialises Evidence Bundle: raw+normalized event refs, integrity hashes, source metadata, transformation history, trust scores |
| 3 | **Confidence** | Evidence Engine | Confidence score per evidence element based on source trust + transformation losses |
| 4 | **Policy** | Policy Engine (via Kernel) | Which AI tier is permissible for this classification, this tenant, this evidence set |
| 5 | **Risk** | Runtime Risk Engine (E-14) | Dynamic risk score for the case + adaptive response ceiling |
| 6 | **AI Reasoning** | AI Engine | Composes prompt (with redaction manifest), routes to L1/L2/L3 per Policy + Risk, produces *DraftRecommendation* with citations, alternatives, false-positive/negative reasoning, and recovery advice |
| 7 | **AI Safety Layer** | AI Safety sub-engine (in AI Engine, described in ARCH-14/B.3) | Policy Validation, Evidence Validation, Confidence Evaluation, Data Classification, Output Validation, Risk Classification, Hallucination Detection, Prompt Injection Detection |
| 8 | **Multi-Engine Adjudication** | Kernel-orchestrated composition of Policy + Risk + Evidence + Identity + Recovery + Audit + AI | Produces *DecisionRecord* — allow / hold-for-approval / reject — with the reasons from each contributing engine |
| 9 | **Business Impact** | Risk Engine + Recovery Engine | Blast-radius estimation on the recommendation itself; e.g., "if analyst acts on this, what changes?" |
| 10 | **Recovery Impact** | Recovery Engine | Reversibility, prerequisites, RTO impact |
| 11 | **Approval Requirement** | Kernel (from Policy) | None / consent / 4-eyes / break-glass |
| 12 | **Execution** | Case Engine | Publishes *ValidatedRecommendation* to `ai.validated.{tenant}` after Approval satisfied. In v0 this is only *presented to a human*; the human is the execution surface. Automated execution is Phase 3 (Response Engine). |
| 13 | **Verification** | Case Engine + Evidence Engine | Confirms the recommendation as displayed matches the signed DecisionRecord; records what the human ultimately chose |
| 14 | **Audit** | Kernel → Audit Engine | Layer-A entries for every stage; Layer-B batching |
| 15 | **Continuous Monitoring** | Risk Engine + Detection Engine | Tracks outcome; adjusts trust scores and confidence adjustments for the next request (Adaptive Trust) |

**AI never writes to the case directly.** The `Publish-Validated-Recommendation` capability is exclusive to the AI Safety sub-engine; the `Append-Timeline` capability that touches the case is held only by the Case Engine. Two capability gates and a distinct engine identity separate the AI draft from the case artifact — this is the Zero-Single-Point-of-Trust posture applied to AI.

## 6. Flow F-5 — Case Assembly and Timeline

| # | Hop | Annotation |
|---|---|---|
| 1 | NATS `findings.*` → Case Engine | B-30x · `Read-Findings` · case-scope logic |
| 2 | Case Engine correlates findings to a case | in-process · additive only |
| 3 | Analyst action from UI (Z5) → Kernel → Case Engine | B-03 → B-10 · session-derived capability · `Append-Timeline` |
| 4 | Validated AI recommendation → Case Engine | see F-4 stage 12 |
| 5 | Timeline stored in Postgres via Storage Engine | B-20 · `Write-Case` · signed by Case Engine SVID |

## 7. Flow F-6 — Evidence Export

| # | Hop | Annotation |
|---|---|---|
| 1 | Analyst requests export → Kernel `Mediate(Export-Case)` | B-03 → B-10 · `Export-Case` (one-shot) · policy + risk + approval per policy |
| 2 | Case Engine assembles bundle: case, timeline, findings, evidence refs, chain-of-custody trail, AI decision records, verification proofs | in-process · Evidence Engine provides provenance |
| 3 | Case Engine signs bundle | in-process · `Sign-Bundle` capability (one-shot) · Case Engine signing key |
| 4 | Case Engine encrypts to recipient (if external) | in-process |
| 5 | Delivery via Connector Engine (email / secure share) | B-50/B-54 · consent + audit |
| 6 | Audit Engine records export event with bundle hash | B-12 · Layer-A + Layer-B |

## 8. Flow F-7 — Audit Path (Layer A + Layer B)

| # | Hop | Annotation |
|---|---|---|
| 1 | Any mediated operation → Kernel local integrity ring | in-Kernel · monotonic seq + prev-hash + Kernel signature |
| 2 | Ring → NATS `audit.layer_a.{tenant}` (asynchronously) | B-12 · signed record · Audit Engine consumes |
| 3 | Audit Engine → ClickHouse `audit_a` and PostgreSQL index | B-20/B-21 · append-only |
| 4 | Audit Engine batches → signs → Object Store `audit/layer_b/…` | B-53 · `Sign-Audit-Batch` (one-shot) · immutability policy |
| 5 | Verification tool (any auditor) → reads Layer-A + Layer-B, verifies chain and signatures | B-04 (support-style ingress) or offline · public verification keys |

## 9. Flow F-8 — Backup and Restore

| # | Hop | Annotation |
|---|---|---|
| 1 | Recovery Engine schedule triggers backup | in-Recovery |
| 2 | Recovery Engine → Storage Engine snapshot request | B-10 · `Write-Snapshot` |
| 3 | Storage Engine → PG/CH snapshot artifacts (Z3) | B-20/B-21 |
| 4 | Recovery Engine → Object Store archival | B-24 · separate backup KEK hierarchy · signed + immutable |
| 5 | Restore path (rare) | `Execute-Recovery` (one-shot, 4-eyes, break-glass) · runbook-driven · Recovery Integrity checks (ARCH-15) |

## 10. Flow F-9 — Break-Glass and Support Access

| # | Hop | Annotation |
|---|---|---|
| 1 | Requester UI → Kernel `RequestBreakGlass` | B-03 → B-10 · `Request-BreakGlass` (one-shot) · consent captured |
| 2 | Approval notifications via Connector Engine | B-30 (kernel-mediated) · Notify targets |
| 3 | Approvers → Kernel `ApproveBreakGlass` | B-10 · `Approve-BreakGlass` (one-shot) · multi-party |
| 4 | Kernel mints time-boxed elevated capability set | in-Kernel · session recording enabled |
| 5 | Session activity → Layer-A + separate elevated-session audit stream | Kernel + Audit · high severity by default |

## 11. Classification Propagation Rules (summary; details in ARCH-12)

- Every hop MUST carry the classification header.
- No downgrade permitted without an explicit Policy Engine decision + audit event.
- Redaction is additive and reversible with proper capability; original is preserved.
- Cross-boundary crossings enforce classification ceilings per ARCH-05 §6.

## 12. Independent Architecture Review

### 12.1 Hidden Assumptions

| Assumption | Handling |
|---|---|
| Kafka replay window is sufficient for restart-recovery of Normalization. | Sized in ARCH-16; replay bounded by ClickHouse raw table (permanent) — raw is the ultimate source of truth. |
| NATS ordering is sufficient for finding delivery. | Findings carry `trace_id` and `sequence`; consumer reconciles; ordering is best-effort not authoritative. |
| Storage Engine consumer keeps up with ingest. | Backpressure signal to Connector Engine per F-1; SLO documented in ARCH-16. |

### 12.2 SPOFs

| Finding | Response |
|---|---|
| **F-1.** *Normalization Engine down → normalized topic starves → correlation stops.* | Raw is preserved; normalization can catch up; Detection has a "stale-normalization" alert; single-node topology has warm standby. |
| **F-2.** *Storage Engine consumer for raw table is single-writer.* | Horizontally scalable per tenant partition; per-partition failure isolated; DLQ pattern for poison records. |

### 12.3 Privilege Escalation

| Finding | Response |
|---|---|
| **F-3.** *AI Engine tries to publish directly to `ai.validated.*`.* | The topic ACL binds only the Safety sub-engine's identity; AI Engine cannot publish. Broker-level enforcement backs up the Kernel gate. |
| **F-4.** *Analyst uses `Read-Logs` to construct a de-facto export by copy-paste.* | Read is audited with query-shape anomaly detection; large-result exfil attempts alert; UI throttles bulk copy. |

### 12.4 Trust-Boundary Violations

| Finding | Response |
|---|---|
| **F-5.** *Redaction manifest lost between Normalization and AI Engine.* | Manifest is a signed durable artifact stored with the normalized event (`AS-M-04` reference); AI Engine verifies before use; missing manifest = refuse. |
| **F-6.** *Case bundle export includes over-classified content unintentionally.* | Export path composes with Policy Engine + Evidence Engine; classification checked per element before inclusion; bundle carries a manifest of included classifications. |

### 12.5 Bottlenecks

| Finding | Response |
|---|---|
| **F-7.** *AI Safety sub-engine becomes a serial bottleneck on F-4.* | Scaled horizontally per tenant partition; parallel per-request; per-tier latency budgets in ARCH-14/B.3. |
| **F-8.** *Audit topic pressure during ingest bursts.* | Layer-A per Kernel-instance local ring absorbs bursts; NATS carries signed refs, not full payloads for high-volume events; batch summarization for `capability.verified`. |

### 12.6 Supply Chain Risks

| Finding | Response |
|---|---|
| **F-9.** *Compromised OCSF-mapping library alters normalization silently.* | Raw preserved (Immutable Event Sourcing); `normalization_version` recorded; re-normalization possible; property tests compare mapping outputs across versions on a fixture corpus per release. |
| **F-10.** *Kafka client library backdoor.* | Message signing beyond mTLS (D-09-2) mitigates payload tampering; hash-pinned deps; ARCH-14/B.2. |

### 12.7 AI-Specific Risks

| Finding | Response |
|---|---|
| **F-11.** *Prompt-injection in event content diverts AI reasoning stage.* | See ARCH-03 §5 + AI Safety Layer (ARCH-14/B.3). Content-quarantine, citations, output validation. |
| **F-12.** *AI produces plausible but hallucinated `raw_ref` values that pass shallow checks.* | Evidence Engine verifies every citation's existence and integrity hash before Safety Layer approves. |

### 12.8 Operational Risks

| Finding | Response |
|---|---|
| **F-13.** *Analysts see so many stages that recommendations feel opaque.* | UI collapses stages by default; expand for audit; "why this" surface always accessible. |
| **F-14.** *Backpressure signaling misconfigured, sources DoS themselves.* | Per-source rate ceilings; graceful drop with counters; documented sizing. |

## 13. Adversarial Architect Review

Assuming unlimited patience, supply-chain and insider compromise, prompt injection, credential theft, and partial infra compromise:

| Attack path attempted | Design response |
|---|---|
| **A-1.** Compromise the AI Engine's signing key and forge *ValidatedRecommendation* records. | AI Engine's key cannot sign `Publish-Validated-Recommendation`; that topic ACL binds to the Safety sub-engine's separate identity; forging requires compromising two engines *and* the Kernel-issued capability. |
| **A-2.** Poison the OCSF mapping library so a specific attacker signature is dropped from normalization. | Raw table preserved and independently readable; Detection can be re-run against raw; diff detection between two normalization versions on a signed fixture corpus reveals silent drop. |
| **A-3.** Insider at cloud provider silently exfiltrates ClickHouse blocks. | Per-tenant DEK envelope encryption (D-04-2); block exfiltration yields ciphertext; audit access to KMS provides a trip-wire; egress anomaly detection on the compute node itself alerts. |
| **A-4.** Steal a `Publish-Ingest` credential and inject synthetic events to bury a real event. | Per-source rate + volume anomaly alerts; source integrity hash bound to source cert prevents payload tampering; Detection Engine's baseline flags volume spikes. |
| **A-5.** Prompt-inject the case timeline to cause a false recommendation. | Timeline content is untrusted in AI paths (F-11); citations must resolve; Evidence Engine verifies; Multi-Engine Adjudication requires Policy + Risk + Evidence + AI concurrence. |
| **A-6.** Plugin compromise attempts to read raw events across tenants. | Plugin holds only Kernel-granted capabilities; no `Read-Raw` unless explicitly granted; broker ACL enforces; egress network policy blocks external exfiltration. |
| **A-7.** Prompt-inject via a *support ticket* uploaded by a customer. | Support ingress goes through the scrubber; secret patterns removed; PII redaction; support content is not a first-class evidence source in AI paths. |
| **A-8.** Kernel compromise mints a broad capability to attacker-controlled identity. | Zero SPOT (ARCH-13): Kernel decisions are shadow-verified by an independent path (see ARCH-13 watchdog architecture); anomalous grants alert; Audit Engine's independent verification of Layer-A catches divergence. |

## 14. Attacker's First-Target Analysis and Redesign

**"If I were an experienced attacker, what part of this design would I target first?"**

**The Multi-Stage Decision Pipeline in Flow F-4 stages 6–7 (AI Reasoning + Safety Layer).** Reason: it is the newest, most complex, most novel component. Attackers historically target novel choke points because the tooling to detect abuse there lags the tooling for older components. A prompt-injection that traverses stage 6, is missed by stage 7, and produces a plausible *DraftRecommendation* referencing *real* evidence is the highest-leverage attack because a downstream human is likely to click accept.

**Redesign response.** The original F-4 stage 7 was described as a serial gate. That's an attractive target for an attacker: bypass one gate, win. Revisions applied to this document:

1. **Consensus decoupling (added F-4 stage 8 — Multi-Engine Adjudication).** The Safety Layer is no longer the sole gate. Its output must concur with independent decisions from Policy, Risk, Evidence, Recovery, and Audit engines. Disagreement escalates to human review with all engines' rationales presented. An attacker who defeats Safety alone gets nothing.
2. **Citation verification is Evidence-Engine-owned (F-12), not Safety-owned.** Prevents the Safety Layer from being both judge and jury on citation truthfulness. Two engines must both trust the citation.
3. **Broker-ACL enforcement of `Publish-Validated-Recommendation`** binds to a distinct identity that AI Engine cannot assume. Attacker must compromise two independent identities plus a Kernel capability grant.
4. **Continuous Monitoring (F-4 stage 15)** updates trust scores based on human accept/reject and post-hoc outcome. A prompt-injection that produces plausibly-benign recommendations that later prove false is caught by trust-score decay on the source; that decay tightens future policy on that source category — a form of adaptive learning without giving AI autonomy.

Repeated the analysis on the redesigned flow. Second-target candidate: the *Kafka broker ACL* update path (if compromised, F-3 could be bypassed). Mitigation escalated to ARCH-13 (Kernel Self-Verification) — broker ACL changes are Kernel-mediated *and* independently verified by the Audit Engine's watchdog on the config-diff stream.

No further high-impact target identified in F-1 through F-9 at the current level of abstraction.

## 15. Decisions

### D-10-1. Multi-Engine Adjudication as the AI-execution gate (not Safety Layer alone)

- **Advantages.** Distributes trust across independent engines; eliminates a single novel gate as an attack surface; produces richer explainability output (each engine's rationale).
- **Disadvantages.** More latency on the F-4 path; more engineering surface.
- **Security Impact.** *Strongly positive.* Zero SPOT applied to AI decisions.
- **Performance Impact.** *Moderate.* F-4 latency ~+200-500ms; acceptable given the human-in-the-loop model.
- **Operational Complexity.** *Higher* — but engines already exist independently.
- **Maintainability.** *Positive* — cleaner separation of concerns.
- **Scalability.** *Positive.*
- **Alternative Designs.** *Safety Layer alone.* Concentrates trust; rejected by adversarial review. *Two-model consensus only.* Solves hallucination but not policy/risk; retained but insufficient alone.
- **Reason for Final Selection.** Directly implements mandates #3 (Distributed Trust) and #4 (Multi-Stage Decision Pipeline).

### D-10-2. Immutable raw persistence *before* ack to source

- **Advantages.** Guaranteed evidence preservation; enables re-normalization; enables detection replay.
- **Disadvantages.** Slightly higher ingest latency.
- **Security Impact.** *Strongly positive.*
- **Performance Impact.** *Small.* Kafka write is ms-scale; acceptable.
- **Operational Complexity.** *Neutral.*
- **Maintainability.** *Positive.*
- **Scalability.** *Positive.*
- **Alternatives.** *Ack-then-persist.* Loss on crash; unacceptable for security telemetry.
- **Reason.** Immutable Event Sourcing is a first principle.

## 16. Open Questions

- Q-10-1. Exact per-tenant partition strategy for Kafka topics — resolved in ARCH-16.
- Q-10-2. NATS vs Kafka for `findings.*` — currently NATS (control-plane) but could move to Kafka if replay is needed. Resolved in Phase B.3.
- Q-10-3. UI expand/collapse of pipeline stages default. Resolved in Phase D.

## 17. Change Log

- **0.1 (2026-07-10)** — Initial draft after two-reviewer discipline.