# Operate Through Compromise: incident runbook

Draft educational template, reviewed against documentation on 2026-10-26. A separate disposable Azure lab exercised worker/peer canary requests, network restriction and restoration on 2026-10-27 (Sydney); it did not establish universal isolation, identity revocation or production recovery. See the companion reader toolkit's verification report. These reusable fields remain empty: `NOT_RUN` is not evidence of success. This is an adaptable decision aid, not an approved production procedure.

Copy into a restricted incident workspace before use. Do not commit real resource identifiers, customer data, tokens, secrets, query results, or incident evidence to this repository. Store evidence separately under the organization's retention and access policies; use references here.

For a bounded reader exercise, use the companion reader toolkit (`README.md` in this folder or the extracted ZIP). It contains an offline decision/evidence validator and an optional disposable Azure lab. The executable checks do not certify this runbook or authorize incident actions.

## Operating boundary

| Field | Complete before execution |
|---|---|
| Incident / exercise ID | |
| Incident commander / alternate | |
| Service owner / critical business function | |
| Evidence custodian / restricted evidence location | |
| Authorized tenant, cloud, subscription, region | |
| Allow-listed source(s), destination(s), operation(s) | |
| Actual SKU, API/CLI/SDK version, telemetry schema | |
| Read permissions / separately approved write permissions | |
| Approval reference / action owner / observer | |
| Window, probe count/rate limit, observation budget | |
| Essential service-health measure / abort threshold | |
| Known-good administrative path / recovery operator | |
| Baseline version / safe rollback version | |
| Test-resource inventory / cleanup owner | |

Start read-only. No production change without service-owner and incident/change-authority approval. No exploitation, credential theft, persistence creation, broad scanning, or destructive actions. Stop if scope, identity, administrative integrity, evidence custody, or recovery access cannot be established. Escalation can be the correct outcome.

Separate evidence classes (`DOCUMENTED`, `OBSERVED`, `ILLUSTRATIVE`, `UNKNOWN`) from test statuses (`NOT_RUN`, `PASS`, `FAIL`, `INCONCLUSIVE`). `PASS` means the predeclared expected outcome was observed with attributable evidence; an expected denial can pass. Missing permission or a timeout is not automatically an expected denial. Risk acceptance never changes a technical result.

## 1. Triage worksheet

Use the first 30 minutes as a proposed decision checkpoint, not an SLA. Record both UTC event time and collection time. Preserve uncertainty rather than completing every box by inference.

| Item | Evidence reference / timestamp | Class / confidence | Owner / unresolved question |
|---|---|---|---|
| Affected workload and process | | UNKNOWN | |
| Selected principal and alternate credentials | | UNKNOWN | |
| Effective direct/inherited grants | | UNKNOWN | |
| Service-mediated authority / other identity attachments | | UNKNOWN | |
| Required business operation and criticality | | UNKNOWN | |
| DNS path, destination, protocol, address family | | UNKNOWN | |
| Route/enforcement observations | | UNKNOWN | |
| Relevant administrative change timeline | | UNKNOWN | |
| Workload/service telemetry coverage and gaps | | UNKNOWN | |
| Authority able to undo containment | | UNKNOWN | |
| Evidence at risk of loss / preservation decision | | UNKNOWN | |

**Handoff:** What is confirmed? What remains inferred? Which unknown changes the containment decision? Who owns the next observation? Record approved urgent action even if evidence could not first be preserved.

## 2. Containment decision record

1. Is activity within approved incident scope? If not, stop/escalate.
2. Is there a trusted administrative/recovery path? If not, escalate to incident command; do not delegate recovery to the suspect workload.
3. What consequence can occur during the next decision interval? Record evidence confidence and cost of delay.
4. Compare proposals against the same threatened operation, consequence of delay and essential dependencies. Reference the organization's reviewed control procedures rather than inventing a new response mechanism.
5. Does the chosen action have an owner, approval, test, service-impact bound and recovery path? If not, escalate or choose another approved action.
6. Apply only the approved bounded change, verify it, then retain, narrow, escalate or recover on evidence.

Choose `ACT`, `HOLD` or `ESCALATE` for each proposal. HOLD defers only the named action; it needs an evidence owner, accepted delay risk, actual UTC deadline and immediate reassessment triggers. ESCALATE names the accountable recipient and alternate. Neither state overrides established emergency procedures.

| Field | Complete for each decision; preserve superseded versions |
|---|---|
| Decision ID / supersedes / actual UTC time | |
| Threatened operation / source / principal / destination | |
| Evidence references, confidence and unknowns | |
| ACT, HOLD or ESCALATE / exact proposal and scope | |
| Approver / service owner / action operator / observer | |
| Accepted service-impact bound / consequence of delay | |
| Rejected alternatives and reasons | |
| Required and prohibited observation / expected result | |
| Evidence-gap owner / trusted recovery operator | |
| Review UTC deadline / escalation recipient and alternate | |
| Immediate reassessment / abort / safe rollback trigger | |
| Actual result / evidence / next decision ID | NOT_RUN |

**Illustrative walkthrough, not a result:** D1 acts on a reviewed worker-local response while holding the shared-grant proposal pending identity evidence. A harmless established session still performing the unwanted operation is a trigger for D2 and incident-command reassessment. This example neither proves D1 worked nor authorizes a broader action. No live times or measurements are supplied.

## 3. Containment watch checklist

This watch sustains an approved response; it does not select an isolation design or authorize service restoration. Define the watch owner/alternate, handover acceptance, actual UTC review deadline, maximum accepted observation gap and escalation recipient before use.

| Watch item | Approved state / evidence and time anchors | Change or gap trigger / action owner | Status |
|---|---|---|---|
| Active decision / configuration version | | | NOT_RUN |
| Prohibited operation / same source, principal, name, peer, port and direction | | | NOT_RUN |
| Fresh session / harmless established session, recorded separately | | | NOT_RUN |
| Healthy dependency / evidence export | | | NOT_RUN |
| Expected management and service evidence / collection gap budget | | | NOT_RUN |
| Reconciliation job / desired-state version / authority | | | NOT_RUN |
| Exception source, destination, identity, protocol, purpose, owner and expiry | | | NOT_RUN |
| Independent administrative/recovery operator and harmless check | | | NOT_RUN |
| Shift handoff / named acceptance / next review | | | NOT_RUN |

**Classify before correcting:** no comparable initial denial means residual access or an inconclusive result, not proven reopening. A deployment proposal is not an applied change. A later configuration diff and changed comparable request support a changed state/outcome; actor and causation need their own evidence. Missing logs never imply success.

Preserve event time, collection time, relevant configuration versions and observation sources. Any scoped pause/override of an owned pipeline needs incident and automation-owner approval, a resumption plan and verification. Never disable all deployments or create an uncontrolled counter-reconciler as a default response.

If recovery access fails or is unknown, escalate through a trusted alternate; do not globally remove containment to regain access. Closure requires approved safe desired state, verified resumed automation, disposition of temporary exceptions, and fresh required/prohibited tests. Reversal must not restore known-unsafe authority or connectivity.

## 4. Identity recovery checklist

| Surface | Inventory / affected dependants | Approved action and exact scope | Verification / timing caveat | Evidence / status / owner |
|---|---|---|---|---|
| Secrets and certificates on application identities | | | | NOT_RUN |
| Managed identity attachments / other attached hosts | | | | NOT_RUN |
| Already issued tokens / active application sessions | | | | NOT_RUN |
| Federation issuer, subject, audience and upstream issuer control | | | | NOT_RUN |
| Direct / inherited RBAC and service authorization | | | | NOT_RUN |
| Group/app permissions / alternative credential paths | | | | NOT_RUN |
| Pipelines, automation and delegated service access | | | | NOT_RUN |
| Owners/admins capable of restoring unsafe trust | | | | NOT_RUN |
| Known-good workload rebuild / minimum reauthorization | | | | NOT_RUN |

Do not store token values or secrets. Record non-secret references and issuer/target context. Distinguish stopping new token issuance from rejecting an existing token and ending a service session. Do not use a human sign-out mechanism as a universal workload revocation step. Choose read-only inspection and service-specific canary checks before approving any identity write. Recovery may need a separately approved known-good workload instead of restoring the suspect one.

## 5. Recovery sequence

Recovery restores an approved business capability, not necessarily the original configuration.

| Stage | Preconditions and approved change | Required positive test | Prohibited-path / guardrail test | Owner / approval / evidence | Status |
|---|---|---|---|---|---|
| Trusted control and observation | | | | | NOT_RUN |
| Known-good workload and minimum identity | | | | | NOT_RUN |
| DNS/resolver dependency for approved service | | | | | NOT_RUN |
| Route and enforcement for exact required flow | | | | | NOT_RUN |
| Service operation and healthy dependency | | | | | NOT_RUN |
| Narrow/remove emergency exception if justified | | | | | NOT_RUN |

This order is a starting dependency model, not a universal Azure sequence. If telemetry needs DNS/networking first, authorize only that bootstrap dependency with its own guardrail. Never remove all emergency controls merely to obtain a health check.

At each stage record DNS answer and client-used peer separately, intended/observed route evidence, fresh-session behavior, service response, application health and observation time. Stop on unexpected destination, failed negative control, degraded essential service, contradictory evidence, or loss of trusted recovery access. Preserve a minimum safe state and escalate.

**Exit and cleanup:** service owner accepts restored capability; incident commander accepts residual risk and monitoring plan; evidence custodian confirms retention; configuration owner reviews drift; temporary rules/grants/test records have expiry or removal evidence; required and prohibited paths are retested. Unexecuted cleanup remains `NOT_RUN`, not “complete.”