# AEGIS — Security Metrics Standard

- **Document ID:** ARCH-24
- **Phase:** C — Standards & Operations
- **Status:** Draft for review (post four-reviewer discipline)
- **Version:** 0.1
- **Date:** 2026-07-10
- **Owner:** Chief Security Architect
- **Depends on:** ARCH-02, ARCH-11 through ARCH-23
- **Consumed by:** every operational review, every audit, every executive brief.

---

## 1. Purpose

Codify the measurable goals of AEGIS's security posture. If a security property matters, it is measured; if it is measured, it is reported; if it is reported, it drives decisions. Implements mandate #31 (Security Metrics) — security must be measurable.

Metrics do not replace judgment. They **inform** judgment and expose drift.

## 2. Requirements

- **REQ-1.** Every metric has: definition, measurement mechanism, unit, baseline, target, SLO, owner, cadence, consumers.
- **REQ-2.** Every mandate in CLAUDE.md § Additional / B / B.3 / C mandates has at least one metric that would show if that mandate is being upheld.
- **REQ-3.** Metrics are collected in Telemetry Engine (E-10) and, for security metrics, in Audit Engine (E-08). Cross-substrate consistency verified.
- **REQ-4.** Metrics that describe tenant-level properties are per-tenant; platform-level metrics are aggregate.
- **REQ-5.** Metrics feed into scheduled reports (weekly ops review, quarterly risk review, annual audit) and into runbooks (drill triggers).

## 3. Metric Catalog

### 3.1 Detection Metrics

- **M-DET-01. Detection Latency (p50/p95/p99).** Time from `observed_at` on ingest to first Finding published for that event class. Owner: Detection Engine. Target: p95 ≤ 2 s single-node baseline (per ARCH-02 NF-2). Alarm on 2× drift from baseline.
- **M-DET-02. False Positive Rate per Rule.** Findings marked FP by analyst / total Findings per rule, rolling 30 d. Owner: Detection Engineering. Target: per-rule ≤ 20% (baseline; tuned per rule). Alarm on step change.
- **M-DET-03. False Negative Estimate.** Missed detections from red-team scenarios and post-incident reviews / total scenarios. Owner: Detection Engineering. Baseline via known synthetic corpus; target trending down.
- **M-DET-04. Rule Coverage.** Percent of MITRE ATT&CK techniques the ruleset addresses. Owner: Detection Engineering. Reported trending; target coverage per priority technique tier.
- **M-DET-05. Rule Deployment Time.** Time from rule authored to deployed to production. Owner: Detection Engineering. Target: ≤ 24 h for high-priority, ≤ 7 d normal.

### 3.2 AI Metrics

- **M-AI-01. Recommendation Accuracy.** Analyst-marked "accepted as posed" / total ValidatedRecommendations, per tier, per classification. Owner: AI + Safety. Target trending up; alarm on step-down.
- **M-AI-02. Hallucination Rejection Rate (Safety Layer V-7).** Rejected-for-hallucination / total DraftRecommendations. Owner: Safety Layer. Baseline set post-v0; alarms on drift.
- **M-AI-03. Prompt-Injection Detection Rate (V-8).** Confirmed / suspected / total DraftRecommendations. Owner: Safety Layer. Baseline + drift alarm.
- **M-AI-04. Redaction Canary Success.** Synthetic secrets planted → surface in logs = fail. Owner: Redaction pipeline. Target: 100%; any failure is SEV-2.
- **M-AI-05. Tier Fallback Rate.** L3 → L2 → L1 fallbacks / total requests. Owner: AI Engine. Anomaly-alerted.
- **M-AI-06. Cost per Tenant per Tier.** Reported to tenant + capped by budget. Owner: AI Engine.
- **M-AI-07. Provider Trust Vector.** Per-provider trust dimensions (§ARCH-11 §6). Owner: Adaptive Trust framework.
- **M-AI-08. Explainability Bundle Completeness.** Fraction of recommendations with all 8 explainability answers structurally present. Owner: Evidence Engine + Safety Layer. Target: 100%; below-100% is a bug.

### 3.3 Evidence and Reproducibility Metrics

- **M-EV-01. Evidence Reproduction Success Rate.** Random-sample reproductions per week / attempts. Owner: Evidence Engine. Target: ≥ 99% for L1 deterministic; documented per-tier for L2/L3.
- **M-EV-02. Citation Resolution Rate.** Citations that resolve to a real Fact / total citations. Owner: Evidence Engine. Target: 100% (Safety Layer refuses otherwise).
- **M-EV-03. Transformation Ledger Consistency.** Weekly dual-store consistency check. Owner: Evidence Engine. Target: 100%; any inconsistency is SEV-1.

### 3.4 Integrity and Audit Metrics

- **M-AUD-01. Layer-A Chain Gap Rate.** Chain gaps detected per Kernel-instance-month. Owner: Audit Engine + Kernel. Target: 0 unexplained gaps; any gap is SEV-1.
- **M-AUD-02. Layer-B Batch Signing Latency.** Median + p99 time from Layer-A record to Layer-B batch signed. Owner: Audit Engine.
- **M-AUD-03. Integrity Beacon Missing Rate.** Kernel-instance-hours without expected beacon / total. Owner: Kernel + Audit. Alarm > threshold.
- **M-AUD-04. Watchdog Alarm Rate.** Alarms per watchdog per month, by watchdog and by severity. Owner: Audit + Policy + Identity + Evidence (as watchdogs).
- **M-AUD-05. Independent Verification Rate.** Fraction of Layer-B batches independently verified by a third-party tool. Owner: Auditor role. Target: 100% by v1.

### 3.5 Capability and Access Metrics

- **M-CAP-01. Capability Denial Rate.** Denials / total verifications, per capability class. Owner: Kernel. Trending; anomaly-alerted.
- **M-CAP-02. Break-Glass Frequency.** Break-glass requests per tenant per week; per role. Owner: Kernel + Identity. Anomaly-alerted; sustained high rate = review.
- **M-CAP-03. Elevated-Session Duration.** p50/p95/p99. Owner: Kernel.
- **M-CAP-04. Privileged Access Attestation Currency.** Fraction of privileged accounts with attestation ≤ 7 days old. Owner: Identity + Ops. Target: 100%.
- **M-CAP-05. Standing Capability Fraction.** Standing vs. break-glass mix. Owner: Ops. Trending — attention to standing-privileged buildup.

### 3.6 Recovery and Availability Metrics

- **M-REC-01. Recovery Time (RTO).** Time from initiation to full-service restore, per drill. Owner: Recovery Engine + Ops. Target: ≤ 4 h (per ARCH-02 NF-1.2). Alarm on regression.
- **M-REC-02. Recovery Point (RPO).** Time between last durable state and incident, per drill. Owner: Recovery. Target: ≤ 15 min control plane; ≤ 5 min audit; ≤ 1 h event plane (per NF-1.3).
- **M-REC-03. Backup Success Rate.** Successful backups / scheduled backups. Owner: Recovery. Target: 100%; any fail is SEV-2.
- **M-REC-04. Restore Drill Pass Rate.** Passed drills / total drills. Owner: Recovery + Ops. Target: ≥ 95%.
- **M-REC-05. Recovery Integrity Attestation Pass Rate.** Attestations with all 5 questions cleanly answered / total. Owner: Recovery + Audit. Target: 100%.
- **M-REC-06. Backup KEK Anomaly Rate.** Anomalous KEK operations per month. Owner: KMS + Recovery.

### 3.7 Plugin Metrics

- **M-PLG-01. Plugin Isolation Violations.** Sandbox escapes, unexpected egress, capability over-use. Owner: Plugin Engine. Target: 0 confirmed violations; any confirmed = SEV-1.
- **M-PLG-02. Plugin Kill-Switch Time.** Time from decision to full termination. Owner: Plugin + Kernel. Target: ≤ 60 s.
- **M-PLG-03. Publisher Trust Distribution.** Distribution of plugin fleet across publisher tiers. Owner: Ops.

### 3.8 Configuration Metrics

- **M-CFG-01. Drift Detection.** Drift events per week (declared vs. effective config). Owner: DriftMonitor. Target: 0; any = SEV-2.
- **M-CFG-02. Ceremony Completion Time.** Time to complete ceremony operations. Owner: Ops. Anomaly-alerted (see ARCH-22 self-critique S-6).
- **M-CFG-03. Configuration Rollback Rate.** Rollbacks per month; per class. Owner: Ops. Trend up = review process.
- **M-CFG-04. Invariant-Set Consistency.** Verified at load and periodically. Target: 100%.

### 3.9 Operational Metrics

- **M-OPS-01. Runbook Drill Freshness.** Fraction of runbooks with drills ≤ 90 d old. Owner: Ops. Target: ≥ 95%; alarm below 80%.
- **M-OPS-02. Post-Mortem SLO.** Post-mortems within 5 business days / total incidents. Target: ≥ 90%.
- **M-OPS-03. On-Call Health.** Rotation load per person; anomaly-alerted (per ARCH-23 M-34).
- **M-OPS-04. Vulnerability Remediation SLO.** Fraction of vulnerabilities patched within SLO by severity (per ARCH-19 §7.2). Target: ≥ 95%.
- **M-OPS-05. Exception Register Trend.** Count of active exceptions; expiry-approaching alerts.
- **M-OPS-06. Vendor Exit-Path Drill Currency.** Fraction of critical vendors with drilled exit within last 12 months. Target: 100%.

### 3.10 Threat-Landscape Metrics

- **M-TL-01. Alarm Distribution Drift.** Distribution of alarm classes over rolling windows; drift = ARCH-21 M-2a.
- **M-TL-02. Emerging-Technique Coverage.** New MITRE technique published → time to rule coverage. Owner: Detection Engineering.
- **M-TL-03. Incident Category Distribution.** Trend of SEV distribution by month. Owner: Ops + Auditor.

## 4. Mandatory Rules

- **M-1.** Every listed metric has a Prometheus/OpenTelemetry recording rule; source of truth is code, not the doc.
- **M-2.** Every alarm-configured metric has: warning threshold, critical threshold, runbook link.
- **M-3.** Metric definitions are versioned; changes go through the config workflow (ARCH-22).
- **M-4.** No sensitive information (PII, secrets, evidence content) in metric labels or values.
- **M-5.** Cross-substrate consistency: security metrics recorded in both Telemetry Engine and Audit Engine (structured summaries) must reconcile; discrepancies are alarms.
- **M-6.** Metrics are per-tenant where applicable; per-tenant SLO reporting for tenant admins.
- **M-7.** Baseline setup after first 30 days of operation for a new deployment; alarms tuned progressively.
- **M-8.** New metric additions require justification, ownership, and consumer identification; drift into vanity metrics forbidden.

## 5. Recommended Practices

- **R-1.** Prefer few well-tuned alarms over many noisy ones.
- **R-2.** Prefer trending views alongside instantaneous values.
- **R-3.** Prefer distributions (histograms) over averages for latency.
- **R-4.** Prefer named alarm severities aligned with IR (SEV-1..4) so on-call knows what to expect.
- **R-5.** Prefer runbook links in alarm annotations.

## 6. Reporting Cadence

- **Weekly.** Ops review dashboard: M-OPS-*, M-DET-01/02, M-AUD-*, M-CAP-*, M-REC-01/02, M-CFG-01.
- **Monthly.** Risk review: trends across all metric classes; correlated with incident + drill outcomes.
- **Quarterly.** Executive brief: high-level posture + KPI trend + top risks.
- **Annually.** Audit-facing report + external audit evidence package.
- **Per-incident.** Post-mortem includes relevant metric snapshots.
- **Per-drill.** Drill report includes M-REC-04, M-OPS-01.

## 7. Verification Process

- **Metric CI.** Recording rules tested; alarm rules tested against fixture data.
- **Metric-to-mandate map.** CI check: every mandate in CLAUDE.md has ≥ 1 associated metric. Missing = build fails.
- **Cross-substrate reconciliation.** Weekly job.
- **Auditor review.** Metrics dashboards accessible to Auditor role; sanity checks on distribution shapes.

## 8. Operational Guidance

- **Alarm tuning:** driven by weekly ops review; noisy alarms tracked as an anti-metric.
- **Dashboards:** organized per stakeholder — Ops, Auditor, Executive, Tenant Admin.
- **Escalation:** SEV-* alarms integrate with the on-call rotation (ARCH-21).

## 9. Future Evolution

- **Continuous-compliance metrics** — automated evidence collection for SOC 2 + ISO 27001.
- **Comparative benchmarks** — per-tenant peer benchmarking (opt-in, differential privacy).
- **Predictive metrics** — machine-assisted trend forecasting on key metrics; advisory.
- **Attack-simulation metrics** — coverage of scenarios red-teamed.

## 10. Independent Architecture Review (Reviewer 1)

- **F-1.** *Metric proliferation.* Governance rule M-8 controls new-metric addition.
- **F-2.** *Alarm fatigue.* R-1 emphasis + tuning cadence; noisy-alarm anti-metric.
- **F-3.** *Metrics measured but not used.* Consumer mandate M-8; each metric has an identified consumer.
- **F-4.** *Metric collection cost.* Sampling + downsampling per ARCH-20; bounded cardinality.

## 11. Adversarial Architect Review (Reviewer 2)

- **A-1.** *Attacker suppresses metrics to hide activity.* Cross-substrate reconciliation catches; watchdog quorum verifies audit-based mirror.
- **A-2.** *Attacker floods a metric.* Cardinality caps; rate limits; anomaly detection.
- **A-3.** *Attacker manipulates baselines during quiet periods.* Baselines are trailing windows + external anchor (audit sample); manipulation surfaces as unusually-fast baseline change.
- **A-4.** *Attacker leverages metric-based SLO to induce operator complacency.* Multiple metric sources per property; auditor visibility; qualitative reviews complement quantitative.
- **A-5.** *Insider tunes SLOs to hide performance regressions.* SLO changes are configuration-managed (ARCH-22); audited; ceremony-controlled for critical metrics.

## 12. Operational Reliability Review (Reviewer 3)

- **O-1. Long-term sustainability of metric definitions.** Versioned; deprecation cycles.
- **O-2. Dashboard maintenance.** Ownership per dashboard; review cadence.
- **O-3. On-call ergonomics.** Alarms tuned; runbook links; distinct severity semantics.
- **O-4. Tenant transparency.** Per-tenant SLO views; trust-building.
- **O-5. Auditor accessibility.** Metrics dashboards fit external audit workflow.
- **O-6. Data retention cost.** Aligned with ARCH-20 defaults; class-tuned.

## 13. Self-Critique (Reviewer 4)

- **S-1.** *"Every mandate has a metric"* is a strong claim; some are hard to quantify (culture, blameless post-mortems). Standard falls back on proxies where direct measurement is infeasible. Acceptable but honest.
- **S-2.** *Overreliance on baseline-relative alarms.* Some absolute thresholds needed (e.g., audit chain gaps: threshold = 0). Standard uses both.
- **S-3.** *Metric-driven decisions can produce metric-gaming behavior.* Auditor review + qualitative complement + cultural discipline mitigate; noted as risk not eliminable.
- **S-4.** *Cost per tenant for full metric stack.* Standard specifies bounded cardinality + tiered retention; needs sizing per ARCH-23.
- **S-5.** *Metric definitions in doc vs. code.* Doc could drift. Standard M-1 mandates code as source of truth; doc references the code; drift detected via CI check.
- **S-6.** *Multi-tenant metric attribution.* Careful design to avoid cross-tenant leakage via label values. Standard M-4 says no sensitive info in labels; needs enforcement (semgrep rule).

Revising in place:
- Added **M-4a.** Semgrep rule bans PII-shaped values in metric labels; CI-fails.
- Added **M-1a.** Metric doc/code drift monitor (CI check).

## 14. Attacker's First-Target Analysis and Redesign

**"If I were an experienced attacker, what part of this standard would I target first?"**

**Baselines.** Reason: alarms depend on baselines. If an attacker can shift baselines during a quiet period (or during an initial deployment window when baselines are being set), subsequent malicious activity that fits the "new normal" won't alarm. This is the metric-equivalent of the poisoned build system attack.

**Redesign response.**

1. **Baseline learning is bounded.** Initial baseline window is documented, not perpetual; subsequent updates are constrained (max delta per window) with anomaly-detection meta-alarms.
2. **External anchors.** Baselines cross-check against reference-fixture data (canary corpus) that AEGIS ships and updates.
3. **Baseline changes audited.** Baseline shifts of > policy threshold produce audit events reviewed by Auditor.
4. **Absolute thresholds.** Some metrics use absolute thresholds independent of baselines (e.g., Layer-A gap > 0, plugin isolation violation > 0).
5. **Alarm rate is itself a metric.** Alarm-rate drop is alarmed (ARCH-21 M-2a).

**Second target.** Metric collection pipeline itself. If Telemetry Engine is compromised, all metrics could be forged. Response: Audit Engine mirrors security-critical metrics from independent sources; reconciliation catches divergence.

**Third target.** Dashboard access to hide metrics from view. Response: metric definitions versioned; changes ceremony-controlled; auditor sees raw data.

## 15. Decisions

### D-24-1. Metric-to-mandate map with CI enforcement

- **Advantages.** Every mandate becomes measurable; no vanity metrics; no silent gap.
- **Disadvantages.** Requires up-front investment in map maintenance.
- **Security Impact.** *Strongly positive.*
- **Performance Impact.** N/A.
- **Operational Complexity.** *Moderate.*
- **Maintainability.** *Positive.*
- **Scalability.** *Positive.*
- **Alternatives.** *Ad-hoc metrics.* Rejected — mandate #31 requires measurement.
- **Reason.** Directly implements mandate #31.

### D-24-2. Absolute + baseline-relative thresholds mixed by property

- **Advantages.** Both stability (baseline) and hard limits (absolute).
- **Disadvantages.** More alarm rules to maintain.
- **Security Impact.** Positive.
- **Performance Impact.** N/A.
- **Operational Complexity.** Moderate.
- **Maintainability.** Positive.
- **Scalability.** Positive.
- **Alternatives.** *Baseline only.* Rejected — attacker can drift. *Absolute only.* Rejected — noisy on varying environments.
- **Reason.** §14 first-target response.

### D-24-3. Cross-substrate reconciliation for security-critical metrics

- **Advantages.** Metric forgery detectable; Zero SPOT applied to metrics.
- **Disadvantages.** Extra storage + compute.
- **Security Impact.** *Positive.*
- **Performance Impact.** Small.
- **Operational Complexity.** Moderate.
- **Maintainability.** Positive.
- **Scalability.** Positive.
- **Alternatives.** *Single-substrate.* Rejected.
- **Reason.** Zero SPOT.

## 16. Open Questions

- Q-24-1. Peer-benchmarking privacy design. Roadmap.
- Q-24-2. Continuous-compliance evidence format (SOC 2). Roadmap.
- Q-24-3. Attack-simulation coverage metric — how to score. Roadmap.

## 17. Change Log

- **0.1 (2026-07-10)** — Initial draft after four-reviewer discipline.