# AEGIS — Recovery Architecture

- **Document ID:** ARCH-18
- **Phase:** B.3 — AI, Plugins, Recovery
- **Status:** Draft for review (post three-reviewer discipline)
- **Version:** 0.1
- **Date:** 2026-07-10
- **Owner:** Chief Security Architect
- **Depends on:** ARCH-04, ARCH-06, ARCH-07, ARCH-08, ARCH-09, ARCH-13
- **Consumed by:** Phase C and D

---

## 1. Purpose

Specify the Recovery Engine (E-11) and the **Recovery Domain** — a dedicated, structurally isolated environment for recovery components so that a compromised runtime cannot compromise the ability to recover from that compromise. Implements B.3 mandate #4 (Recovery Isolation) and the earlier Recovery Integrity mandate. Elevates the Recovery Engine from a backup/restore utility to a governed sub-platform with its own trust anchor.

## 2. Guiding Principles

1. **Recovery must survive what it recovers from.** A ransomware event on the runtime deployment must not disable recovery. A Kernel compromise must not disable recovery.
2. **Recovery is verifiable.** Every recovery answers five integrity questions (per elevation mandate #10): complete? evidence preserved? rollback possible? drift introduced? integrity re-established?
3. **Recovery is drilled.** Untested recovery is worse than none — it creates false confidence.
4. **Recovery is boring.** Prefer plain, verifiable mechanisms (immutable snapshots, signed bundles, offline copies) over sophisticated exotica.

## 3. Recovery Domain

### 3.1 Definition

A **Recovery Domain** is a deployment scope containing only:
- Recovery Engine instance(s).
- Recovery-only Kernel replica or a bounded Kernel-recovery-mode profile (§4.3).
- Signed backup catalog store.
- Object-store backup targets (both immutable and air-gap).
- Optional external notification target for Recovery alarms.

### 3.2 Isolation Properties

- **Separate identities.** Recovery Domain SVIDs are issued under a distinct sub-hierarchy of the Kernel intermediate CA. Compromise of a runtime engine SVID does not authenticate to the Recovery Domain.
- **Separate credentials.** Backup KEK hierarchy (AS-K-08) is separate from live-data KEK. Distinct KMS scopes / distinct HSM slots.
- **Separate network.** Recovery Domain runs in its own Kubernetes namespace (or its own network segment on VPS). Network policy: **no ingress from the runtime deployment**; egress *from* the Recovery Domain *to* the runtime is permitted for restore operations only, under Kernel mediation.
- **Separate storage.** Backup object store is a distinct account/bucket, accessible only by Recovery Domain identities. Immutability policy at the storage layer (S3 Object Lock or equivalent).
- **Separate audit stream.** Recovery Domain writes its own audit records; they land in Layer-A + Layer-B like all others, but the signing key is Recovery Domain's own leaf.
- **Optionally physically separate.** Air-gap copy is on write-once media or a network-isolated store; refreshed on schedule.

### 3.3 Runtime → Recovery Data Flow

The runtime pushes backup data *out* through a narrow, mTLS-authenticated, capability-scoped pipeline:
- Backup capability (`Write-Snapshot`) is granted to the Storage Engine's SVID.
- Snapshots are encrypted under the backup KEK hierarchy at the runtime side; ciphertext + metadata push to the Recovery Domain object store.
- Recovery Domain does not need to decrypt for storage; decryption keys are held only for restore under Kernel-mediated `Execute-Recovery`.

### 3.4 Recovery → Runtime Data Flow

Only during a controlled restore:
- Kernel (in Recovery Mode or under an active break-glass) grants `Execute-Recovery` to a Recovery Engine action.
- Recovery Engine reads snapshot ciphertext from backup store, decrypts under the backup KEK hierarchy inside the Recovery Domain, and streams to the destination (rebuilt runtime).
- Every step signed, audited, and verifiable.

## 4. Recovery Engine (E-11) — Specification

### 4.1 Purpose

Own backup, restore, DR runbooks, degraded-mode orchestration, chaos-drill scheduling, rollback tooling, and Recovery Integrity verification.

### 4.2 Responsibilities

1. **Scheduled backup.** For every backup class (control plane, event plane, audit archival, configuration, catalog artifacts) — cadence per policy.
2. **On-demand snapshot.** Triggered by operators or by automated pre-change hooks.
3. **Restore execution.** Both partial (single tenant, single case, single audit range) and full (whole platform).
4. **DR drills.** Scheduled and ad-hoc; exercise restore paths without touching production.
5. **Chaos injection.** Inject synthetic Kernel-integrity or engine-failure events at test tenants to verify watchdogs + alarms + runbooks.
6. **Recovery Integrity attestations.** Every recovery emits a signed attestation answering the five integrity questions (§8).
7. **Ransomware resilience.** Maintain immutable + air-gap copies; watch for backup-target tampering.

### 4.3 Kernel-in-Recovery-Mode

A bounded profile of the Kernel that runs *inside* the Recovery Domain for restore operations:
- Loads its identity from Recovery Domain's SVID hierarchy.
- Its capability vocabulary is restricted to Recovery-related operations (`Execute-Recovery`, `Read-Snapshot`, `Write-Snapshot`, `Verify-Backup`, `Sign-Recovery-Attestation`, `Break-Glass-Recovery`).
- It cannot mint capabilities for the runtime deployment.
- Its decisions are watchdog-verified by an Audit-in-Recovery watchdog running alongside.

### 4.4 Recovery-First (Engine-level)

1. *How does it fail?* Backup miss; restore corruption; slow restore; ransomware on backup target; misapplied runbook; snapshot integrity failure.
2. *Detection.* Backup success metric; watermark of latest archived state; drift from RTO/RPO SLOs; restore-drill pass rate.
3. *Initiation.* Automatic backup; automatic drill; human for real restore.
4. *Automated?* Yes for backup + drills; runbook-driven for real recovery.
5. *Rollback?* Every restore leaves source intact; rollback = point-in-time selection; recovery attestations track drift.

## 5. Backup Strategy

### 5.1 Data Classes and Cadence

| Class | Substrate | Cadence | Retention (default) | Immutability |
|---|---|---|---|---|
| Control plane (PostgreSQL) | Full: daily; Incremental: 15 min | 90 d hot / 7 y warm | Object Lock (compliance mode) |
| Event plane (ClickHouse) | Snapshots + parts backup: hourly; Full: weekly | 90 d hot / 1 y warm | Object Lock (compliance mode) |
| Audit Layer-A ring | Streamed continuously | 7 y | Object Lock (compliance mode) + air-gap copy |
| Audit Layer-B batches | Signed at production; archived immediately | 7 y+ | Object Lock (compliance mode) + air-gap copy |
| Configuration snapshots | On every change; snapshot every 24 h | 1 y + 5 pinned baselines | Object Lock |
| Catalog artifacts (Risk weights, Trust weights, Policy corpus, Prompt templates) | On every ceremony | Indefinite | Object Lock |
| KMS/HSM backup material | Per crypto policy | Per crypto policy | HSM-native |

### 5.2 Encryption

- Per-tenant DEK envelope encryption for runtime data (D-04-2).
- Backup KEK hierarchy (AS-K-08) wraps the DEK snapshots — distinct from live-data KEK.
- **Backup ciphertext does not reveal tenant identity or data structure** — metadata carries only minimal fields necessary for restore.

### 5.3 Ransomware Resilience

- **Immutable storage.** Object Lock at the backup target denies delete/overwrite for the retention window. Even AEGIS admins cannot delete before expiry without breaking storage-side policy.
- **Air-gap copy.** Independent copy on a separate account/region/provider or physical media; refreshed on cadence. Air-gap read-only from AEGIS runtime.
- **Backup-verification canary.** Recovery Engine periodically restores a canary to a scratch environment and verifies integrity; degradation alerts.
- **Backup-of-backup.** Layer-B audit chain enables detection of gaps in the backup sequence itself.

## 6. Restore Workflows

### 6.1 Categories

- **Point-in-time restore of a single tenant** (data loss on customer request or accidental deletion).
- **Point-in-time restore of a data class** (e.g., a corrupted case-metadata rollback).
- **Full-platform restore after catastrophic loss** (rebuild from backups on new infrastructure).
- **Kernel emergency restore** (per ARCH-13 §7).
- **Audit-only restore** (regulator request for a specific window).

### 6.2 Restore Contract

For every restore:
1. Request initiated by an authorized operator via UI or CLI → Kernel `Mediate(Execute-Recovery)`.
2. Multi-Stage Decision Pipeline (ARCH-11) applied: Evidence → Confidence → Policy → Risk (destructive risk high by default) → Business Impact → Recovery Impact (self-referential; forces documented target) → Approval (4-eyes + break-glass).
3. Recovery Engine acts inside the Recovery Domain.
4. Verification pass: hashes, signatures, chain integrity.
5. Handoff to the runtime deployment via Kernel-mediated stream.
6. **Post-restore verification** (mandatory): watchdog engines re-verify Kernel integrity + capability directory + audit chain continuity + configuration parity.
7. **Recovery Integrity Attestation** produced and signed.

### 6.3 Test Restores (Drills)

- **Every backup class** subject to a drill on a cadence:
  - Control plane: weekly drill of a tenant restore.
  - Event plane: weekly drill of a time-window restore.
  - Audit: weekly drill of hash-chain verification + range restore.
  - Configuration: daily drill of rollback.
  - Full-platform: quarterly drill.
- Drills produce Recovery Integrity Attestations too — those attestations are the KPI for readiness.

## 7. Time Integrity in Recovery (Mandate #9)

- Backup timestamps are signed by the Kernel (or Kernel-in-Recovery-Mode) at production; not source-clock trust.
- Monotonic anchoring: every backup carries a monotonic sequence + previous-hash to detect skipped or reordered backups.
- Time confidence recorded per backup: NTP-sync status at production.
- Restore ordering is by monotonic sequence, not wall-clock — defends against clock-skew or backup-reorder attacks.

## 8. Recovery Integrity — Five Questions (Mandate #10)

Every recovery produces a signed Recovery Integrity Attestation:

1. **Was recovery complete?** — All classes restored to target state; residual reconciliation logged; verified by state-hash comparison.
2. **Was evidence preserved?** — Layer-A + Layer-B chain continuity re-verified; no unexplained gap; watchdog engines report clean.
3. **Was rollback possible?** — Pre-recovery state retained; rollback point available; retention window logged.
4. **Did recovery introduce drift?** — Configuration diff between pre- and post-recovery reported; unexpected drift is a high-severity audit event; expected drift is accepted per Policy.
5. **Was integrity re-established?** — Watchdog quorum passes; capability directory reconciled; alarm cleared.

The attestation is signed by the Recovery Engine + counter-signed by the Audit Engine (as watchdog). Distributed Trust applied to recovery.

## 9. Safe Failure Modes (Mandate #7)

| Mode | Trigger | Behavior |
|---|---|---|
| **Normal** | Backups on schedule; drills passing | Automatic backups; scheduled drills |
| **Degraded** | Backup target slow; drill regressions | Backups continue; alerts; catch-up plan |
| **Recovery** | Restore actively in progress | Runtime traffic redirected as needed; explicit progress reporting |
| **Maintenance** | Recovery Engine version upgrade | Backups continue via alternate replica; upgrade drill immediately after |
| **Emergency** | Backup integrity failure; Kernel emergency; watchdog alarm quorum | Halt runtime writes to R/HR; escalate; require operator ack before restore |

## 10. Defensive Telemetry (Mandate #8)

Recovery Engine telemetry is proportional:
- **Always collected:** backup success/failure per class, watermarks, drill pass rates, RTO/RPO SLO tracking, retention health, immutability policy status.
- **On anomaly:** trace-level for a specific backup run, ciphertext-size drift, unexpected metadata.
- **Never collected:** ciphertext content in telemetry; tenant PII beyond what identifies a backup class.

## 11. Independent Architecture Review (Reviewer 1)

### 11.1 Hidden Assumptions

| Assumption | Handling |
|---|---|
| Object Lock is honored by the storage provider. | Verified at provisioning; canary drill attempts to delete an expiring-tomorrow object; failure = alarm. |
| Air-gap copy remains genuinely air-gapped. | Access log at the medium; cadence documented; drill inspects. |
| Restore drill exercises real failure modes. | Chaos-drill scheduler produces realistic failures; retrospectives feed improvements. |

### 11.2 SPOFs

| Finding | Response |
|---|---|
| **F-1.** *Recovery Domain outage → no backups.* | HA within Recovery Domain; degraded-mode continues writes to a temporary target; catch-up on recovery. |
| **F-2.** *Backup KEK compromise.* | KEK hierarchy separate; rotation ceremony; kill-switch drilled. |
| **F-3.** *Kernel-in-Recovery-Mode is itself a SPOF for restore.* | Multiple replicas; Recovery Domain has its own Watchdog quorum. |

### 11.3 Privilege Escalation

| Finding | Response |
|---|---|
| **F-4.** *Runtime engine attempts to read a backup directly.* | Recovery Domain identities differ; access refused; audit alarm. |
| **F-5.** *Insider triggers Execute-Recovery for a non-approved target.* | 4-eyes + break-glass + Multi-Stage Pipeline including Business Impact + Recovery Impact + Post-recovery verification; audit records intent + target. |

### 11.4 Trust-Boundary Violations

| Finding | Response |
|---|---|
| **F-6.** *Runtime and Recovery share a cloud account.* | Documented anti-pattern; recommended topology is separate account/region; guide + drift monitor. |
| **F-7.** *Restore-side decryption keys leak.* | Keys transient; used only inside Recovery Domain; HSM-anchored; never egress. |

### 11.5 Bottlenecks

| Finding | Response |
|---|---|
| **F-8.** *Full-platform restore misses RTO.* | RTO of 4h is baseline; sizing documented; incremental restore first, catch-up second. |
| **F-9.** *Backup catalog scaling.* | Catalog is a normal control-plane table with per-class indexes. |

### 11.6 Supply Chain

| Finding | Response |
|---|---|
| **F-10.** *Object-lock provider compromise.* | Multi-provider air-gap copy; canary detects; Runbook-driven fallback. |
| **F-11.** *Recovery Engine dependency compromise.* | Small dependency set; hash-pinned; RIV monitors; separate signing hierarchy for Recovery. |

### 11.7 AI-Specific Risks

| Finding | Response |
|---|---|
| **F-12.** *AI Engine attempts to include itself as a restore target.* | AI Engine's model artifacts are recoverable, but restore path applies to signed models only; unsigned models rejected. |

### 11.8 Operational Risks

| Finding | Response |
|---|---|
| **F-13.** *Operators mis-select target for a partial restore.* | UX gates + confirmation with target detail + 4-eyes; simulation mode available. |
| **F-14.** *Retention policy changes lose historical data.* | Policy changes ceremony-controlled; retention shortening for existing data requires explicit destructive action with audit. |
| **F-15.** *Drill fatigue.* | Drills scheduled and automatic; results dashboarded; drill failure is treated as an incident. |

## 12. Adversarial Architect Review (Reviewer 2)

| Attack path | Design response |
|---|---|
| **A-1.** Ransomware on runtime spreads to backups. | Recovery Domain separate identity + separate network + immutable storage + air-gap copy. Ransomware inside runtime cannot delete or encrypt backups. |
| **A-2.** Insider triggers restore to attacker-prepared backup. | Backup catalog signed at ceremony; unsigned entries refused; canary drill verifies known-good; multi-stage pipeline includes Business + Recovery Impact review. |
| **A-3.** Cloud compromise reads Recovery Domain object store. | Backup ciphertext encrypted with KEK hierarchy held only in Recovery Domain HSM/KMS; cloud read yields ciphertext without keys. |
| **A-4.** Compromise KMS to obtain backup KEK. | KMS access audited; separate hierarchy; kill-switch drilled; multi-provider option for KMS in ARCH-11. |
| **A-5.** Prompt-inject an operator (via any UI or chat) to accept a rogue restore. | Restore UX presents structured verification (hashes, timestamps, target diff) that a human verifies out-of-band; no natural-language override. |
| **A-6.** Supply-chain compromise of Recovery Engine binary. | Signed under release hierarchy; RIV monitors; watchdog cross-verifies attestations. |
| **A-7.** Slow long-persistence attacker corrupts backups gradually. | Canary drill on rotating tenant sample detects; hash chain of backups makes silent corruption visible. |
| **A-8.** Object Lock provider ignores retention. | Multi-provider air-gap copy; canary tries to delete an object one day before expiry — failure = healthy, success = alarm and switch. |

## 13. Operational Reliability Review (Reviewer 3)

| Concern | Response |
|---|---|
| **O-1. 15-year backup format stability.** | Formats versioned; verifier CLI accepts historical versions; migration documented. |
| **O-2. Real DR muscle memory.** | Drills quarterly minimum; unannounced drills part of rotation; post-drill retrospectives. |
| **O-3. Cost of maintaining Recovery Domain.** | Sized to platform; optimized for low ongoing cost + high ceremony-mode cost. |
| **O-4. Observability of backup health.** | Dashboards + alerts + retention KPIs; SLO for backup success + drill pass rate. |
| **O-5. Upgrade safety of Recovery Engine.** | Canary drills after upgrade; automatic rollback if drill regresses. |
| **O-6. Long-term storage medium evolution.** | Air-gap medium is migrated on cadence (e.g., every 3-5 years); documented refresh process. |
| **O-7. Documentation.** | Runbooks live in-repo alongside the code; auto-tested where possible; auditor-visible. |
| **O-8. Cross-region considerations.** | Region choice per tenant/deployment; documented residency constraints. |

## 14. Attacker's First-Target Analysis and Redesign

**"If I were an experienced attacker, what part of this design would I target first?"**

**The Recovery Domain's KEK hierarchy.** Reason: it is the smallest artifact whose compromise unlocks the largest amount of data. Backups are enormous, encrypted, and *quiet* — no user watches them daily. If an attacker can access the backup KEK, they can decrypt years of data without ever touching the runtime.

**Redesign response.**

1. **KEK never leaves HSM/KMS in cleartext.** All wrapping/unwrapping happens inside the HSM boundary; keys are never exported. Enforced by HSM policy, not just software.
2. **KEK use is Kernel-in-Recovery-Mode-mediated.** Even inside the Recovery Domain, KEK operations require an active Kernel capability, produce audit records, and go through multi-stage approval for wide-scope operations (e.g., decrypting > N tenants requires additional approval).
3. **KEK anomaly detection.** Every KEK operation is telemetered; frequency, scope, and identity are watched for anomalies. Unusual pattern = alarm + kill-switch on the KEK (see #5).
4. **Multi-KEK per tenant class.** Rather than one KEK per platform-wide backup, backup KEKs are scoped per tenant class (or per tenant for HR-eligible tenants). Compromise blast radius bounded by KEK scope.
5. **KEK kill-switch drilled.** Any KEK can be revoked in a documented ceremony; a revoked KEK cannot decrypt future backups; existing ciphertexts are re-wrapped at ceremony under a new KEK during a controlled rotation. Ceremony rehearsed at least annually.
6. **Air-gap copy uses a distinct KEK.** Cloud-KEK compromise does not compromise air-gap ciphertext.

**Second target after redesign.** Operator UI for restore. Response: structured verification of target + hashes + diffs; simulation mode; 4-eyes on any wide-scope restore; audit-visible pre-image.

**Third target.** Object-lock policy at the storage provider. Response: multi-provider air-gap; retention canary drill probes lock enforcement; suspicious deletes alarm and trigger provider switch.

No further high-impact silent-compromise vector identified at this abstraction.

## 15. Security Economics (Mandate #10)

- **Attacker cost raised.** Ransomware needs to reach the Recovery Domain *and* the air-gap copy *and* the KEK — each independently protected.
- **Attacker cost raised.** Silent long-persistence corruption is defeated by canary drills.
- **Defender cost reduced.** Recovery Integrity Attestation is a single artifact for auditor and operator; drills are automated; kill-switch is one Kernel operation.

## 16. Decisions

### D-18-1. Dedicated Recovery Domain with separate identity + network + storage + KEKs

- **Advantages.** Structural survival of ransomware and runtime compromise; recovery credibility.
- **Disadvantages.** More deployment complexity; requires customer discipline (separate accounts).
- **Security Impact.** *Strongly positive.*
- **Performance Impact.** N/A.
- **Operational Complexity.** *Higher* — offset by real recovery capability.
- **Maintainability.** *Positive* — clean separation.
- **Scalability.** *Positive.*
- **Alternatives.** *Same-domain recovery.* Rejected — mandate #4. *Third-party recovery service.* Rejected — trust concentration.
- **Reason.** Mandate #4.

### D-18-2. Immutable backups + air-gap copy + KMS-only KEK usage

- **Advantages.** Ransomware resilience; supply-chain resilience at storage layer.
- **Disadvantages.** Two backup targets to operate.
- **Security Impact.** *Strongly positive.*
- **Performance Impact.** Neutral.
- **Operational Complexity.** *Moderate.*
- **Maintainability.** *Positive.*
- **Scalability.** *Positive.*
- **Alternatives.** *Single storage target.* Rejected. *KEK-in-runtime.* Rejected.
- **Reason.** Ransomware is the top availability threat for security platforms.

### D-18-3. Multi-KEK per tenant class + KEK kill-switch drilled

- **Advantages.** Addresses primary redesign target (§14); blast-radius bounded.
- **Disadvantages.** More keys to manage.
- **Security Impact.** *Strongly positive.*
- **Performance Impact.** *Neutral.*
- **Operational Complexity.** *Moderate.*
- **Maintainability.** *Positive.*
- **Scalability.** *Positive.*
- **Alternatives.** *Single global backup KEK.* Rejected.
- **Reason.** Zero SPOT applied to backup encryption.

### D-18-4. Recovery Integrity Attestation signed by Recovery + Audit (watchdog)

- **Advantages.** Distributed Trust for recovery; auditable; verifiable.
- **Disadvantages.** Two-signature workflow.
- **Security Impact.** *Strongly positive.*
- **Performance Impact.** *Neutral.*
- **Operational Complexity.** *Moderate.*
- **Maintainability.** *Positive.*
- **Scalability.** *Positive.*
- **Alternatives.** *Single signature.* Rejected — Zero SPOT.
- **Reason.** Mandate #10 (Recovery Integrity) + #1 (Zero SPOT).

## 17. Open Questions

- Q-18-1. Precise object-store provider matrix per topology. Resolved in ARCH-19 (Deployment Architecture, Phase D).
- Q-18-2. Backup catalog scaling for very large tenants. Resolved in Phase C.
- Q-18-3. Legal-hold workflow ownership. Resolved in Phase C / ARCH-14 (Supply Chain).

## 18. Change Log

- **0.1 (2026-07-10)** — Initial draft after three-reviewer discipline.