# Operate Through Compromise: reader validation toolkit 1.0.0

**Status:** offline-tested and exercised in an authorized disposable Azure lab, including a corrected-workflow clean replay, in Australia East on 2026-10-27 (Sydney). See the [verification report](azure-lab-verification.md) for observed results, runtime fixes and cleanup evidence. This is not production approval or a universal isolation claim.

This toolkit tests a narrower, falsifiable part of the series thesis: can you record a scoped decision, compare the same operation before/during/after a change, notice when the evidence is inadequate, preserve a healthy peer and hand off the recovery decision?

The offline track needs Node.js 22 or newer and no packages, credentials or cloud access. The optional Azure track uses Azure CLI, its Bicep CLI, PowerShell 5.1 or newer and Python 3 already present on the selected Ubuntu image. It creates billable resources. Do not run it against production or put any real workload/data in the lab.

## Access and evidence boundary

The canonical source is `static/downloads/operate-through-compromise/` in the blog repository. The versioned ZIP and SHA-256 checksum beside it are intended for the site's `/downloads/operate-through-compromise/` path after an approved site deployment. They are not claimed publicly deployed yet. Clone/download this repository now, or use the verified published ZIP once available. Keep the unpacked toolkit together.

Files use the public Azure cloud with one Blob private endpoint. Sovereign clouds, IPv6, alternative endpoint paths, shared identities, federation, existing sessions, real attacker behavior and production recovery are outside this lab. No exploitation, stolen credentials, persistence or unbounded scanning is performed.

`ILLUSTRATIVE` means fictional exercise input. `OBSERVED` is used only by the probe collector for the operation it actually attempted. Neither is authenticated just because it appears in JSON. Never replace the example's class with OBSERVED to manufacture evidence.

## 1. Offline exercise: run it before Azure

From the extracted toolkit directory:

```powershell
node --test .\validate.test.cjs
.\lab.test.ps1
New-Item -ItemType Directory -Path .\private -ErrorAction Stop
node .\exercise.cjs | Set-Content -Encoding UTF8 .\private\exercise.json
node .\validate.cjs .\private\exercise.json
```

The example deliberately returns **INCONCLUSIVE**, a valid record structure, a bounded worker no-response observation and exit code **2**. That is the correct result: a timeout is compatible with containment but does not prove that a network rule caused it. There is intentionally no green "isolation proven" result. Exit **1** means invalid data, a contradictory successful prohibited request, or an execution error; read the output, not just the exit code.

The PowerShell guard tests intercept every Azure CLI call locally. If Python 3 is installed locally, also run `python .\probe_test.py` for mocked token/service-success/error/timeout tests; no network calls are made. Python is needed on the guest for real probes, not for the Node.js offline exercise.

Make separate copies of the fixture. Do not overwrite the original decision history.

| Challenge | Change in your copied JSON | Required learning / observable output |
|---|---|---|
| Scope and comparability | Change contained worker's `principal_id` | INVALID: different identities cannot support the same before/after claim |
| Decision quality | Set state HOLD; remove `gap_owner` or make review earlier than decision | INVALID: a named hold without ownership/deadline is not a completed decision |
| Falsify containment | Change contained worker to HTTP status 200 | CONTRADICTED: an unwanted operation still succeeds |
| Distinguish controls | Change contained worker to HTTP status 403 | INCONCLUSIVE: authorization/service failure does not establish network denial |
| Protect availability | Make contained peer NO_RESPONSE with null status | Missing healthy-peer operation is surfaced, not hidden by worker failure |
| Sustain the boundary | Add an OPEN exception expired at watch time | Expired exception appears as a gap |
| Hand off honestly | Lower the accepted watch-gap budget below the observed interval | Gap budget exceeded; no silent success |
| Preserve evidence classes | Change only the record class to OBSERVED | INVALID: mixed classes; a label cannot turn the exercise into an experiment |

Use the separate incident runbook and tabletop pack for ownership, escalation and discussion. In the ZIP they are included as `incident-runbook.md` and `tabletop-pack.md`; repository originals remain under `btpe/toolkits/operate-through-compromise/`.

The validator enforces required fields, unique six-cell matrix, chronology, source/identity/operation comparability, bounded probe windows, explicit deadlines, exception ownership and evidence-class consistency. It cannot assess whether a human approval is genuine, an impact bound is acceptable or a recovery target is safe.

## 2. Optional Azure experiment: architecture and hypotheses

```text
Trusted operator -- Azure Run Command --> worker VM + peer VM (no inbound/public VM IP)
                                            |          |
                                    separate system-assigned identities
                                            |          |
                       one VNet / private DNS / Blob private endpoint / empty canary
                       worker NIC NSG changes; peer NIC NSG does not
```

Both VMs have container-scoped **Storage Blob Data Reader** grants. The operation is an OAuth-authenticated list of the empty `canary` container; it does not read customer data or write blobs. Storage public network access, shared keys and anonymous Blob access are disabled. A separate endpoint subnet hosts `10.77.2.4`; a linked private DNS zone resolves the normal Blob name to that private endpoint. Outbound NAT is explicit so VM-agent control does not rely on implicit outbound access.

| Phase | Worker fresh request | Healthy peer fresh request | Interpretation |
|---|---|---|---|
| Baseline | HTTP 200 | HTTP 200 | Establish service, identity and network prerequisites before intervention |
| Contained | Typically no response within 15-second service timeout | HTTP 200 | Compatible with a bounded network effect; not proof of a denial or complete isolation |
| Restored | HTTP 200 | HTTP 200 | Lab canary connectivity recovered; not production recovery acceptance |

Containment adds only the worker NIC's `ContainStorageHttps` outbound TCP/443 deny to **10.77.2.4/32**, the lab's dedicated Blob private endpoint. It does not deny the entire Storage service tag: doing so could also break VM-agent result delivery through unrelated Azure Storage. Other containers behind this same endpoint would share the network effect, so keep this account empty except for the synthetic canary. This is not a production prescription.

Fresh Python processes make fresh service requests with `Connection: close`. The probe first requires the Blob name to resolve only to `10.77.2.4` and records its DNS addresses; unexpected answers stop before token acquisition. This DNS observation is not an independently captured client peer or route trace. No existing-session termination is tested. NSG statefulness means existing connections can survive a rule change. Each phase collects exactly one operation per VM, sequentially, not a statistically representative or simultaneous availability measurement.

The lab preserves the same identity grants throughout. It demonstrates network-path intervention and its separation from authority, **not identity revocation or rebuild**. The tabletop exercises the missing shared-identity, surviving-session, drift and recovery branches without pretending those were run.

## 3. Approval, permissions, costs and prerequisites

Use a dedicated test subscription or an explicitly approved sandbox. Obtain an authorized operator, budget owner, exact subscription/region/window, essential-peer expectation, abort decision, independent administration path and cleanup owner. Fill the runbook before cloud execution.

Resource footprint: two Ubuntu 24.04 VMs (default `Standard_B1s`, configurable), two managed OS disks, NAT Gateway, one Standard public egress IP, VNet/two subnets, two workload NICs/NSGs, one Blob private endpoint and its managed NIC, private DNS zone/link, one Standard LRS Storage account and an empty container. **NAT, private endpoint, DNS and IP charges persist even if VMs are stopped.** Disks and storage also persist. Estimate local prices for your selected duration with the Azure pricing calculator; there is no fabricated dollar estimate or zero-cost claim. Delete the dedicated group after the exercise and verify billing inventory.

Check provider registration (`Microsoft.Compute`, `Microsoft.Network`, `Microsoft.Storage`), quota, regional VM/image availability, Azure Policy and actual current API support before approval. The selected public-cloud APIs are pinned in `main.bicep`; compilation does not establish runtime/region support.

The human needs resource-group creation permission at the chosen subscription scope, resource deployment/write/read permissions in the new group, and `Microsoft.Authorization/roleAssignments/write` for the two container-scoped grants. A Contributor alone cannot create those grants. Scope RBAC administration through your organization; no instruction to grant broad Owner access is necessary. Probe invokes privileged VM Run Command and reads deployment outputs. Recheck the action API's required permissions against your current CLI/provider: the linked Linux guide lists `Microsoft.Compute/virtualMachines/runCommands/write` and subscription-level `Microsoft.Compute/locations/runCommands/read`; do not conflate action and managed Run Command permissions. Cleanup requires permission to delete the dedicated group. The workload identity needs only the stated container-scoped reader grant.

Run Command depends on a healthy VM agent, platform networking and a trusted Azure operator/session; it is **not an independent recovery guarantee** or a path suitable for every real compromise. Do not broaden egress or containment to fix failed agent access; stop and escalate. The generated SSH public key does not enable inbound SSH.

Set a maximum lab window (for example, an author-approved 60 minutes), not an unbounded retry loop. This code has no retry-until-green logic. If RBAC propagation prevents baseline 200, retain the failure and retry later into a **new private output directory** within the approved timebox. Do not run containment without a healthy baseline.

## 4. Local compile and deployment

These commands are examples requiring substitution of your own approved values. Use an Azure CLI login in the approved tenant. The runner supplies the subscription explicitly on each Azure call; it does not change the CLI default account.

```powershell
ssh-keygen -t ed25519 -f .\private\otc-key
$subscription = '<approved-subscription-guid>'
$group = 'otc-lab-reader-unique'
$evidence = (Resolve-Path .\private).Path
$common = @{
    SubscriptionId = $subscription
    ResourceGroup = $group
    Location = 'australiaeast'
    SshPublicKeyFile = '.\private\otc-key.pub'
    OutputDirectory = $evidence
}
.\lab.ps1 -Action Plan @common
# Read the template, current docs, pricing and approved inventory BEFORE this:
.\lab.ps1 -Action Deploy @common -Approved
.\lab.ps1 -Action Probe -Phase baseline @common -Approved
```

`Plan` compiles locally; it is not ARM what-if, pricing or permission validation. It requires the public key but creates no group. `Deploy` refuses an existing group, creates/tag-owns a new one and requests ARM validation before provisioning. **Never put anything else in that group.** Every Azure-executing action is explicitly approval-gated. A failed deployment may leave billable partial resources; see cleanup below.

Inspect both baseline JSON files. They must show service HTTP 200. Keep raw JSON local and restricted; it includes lab resource/principal IDs and endpoint names, never tokens or Blob contents. Do not commit the `private` directory or real evidence.

The collector attributes source and principal IDs from the reviewed deployment configuration. It does not independently attest guest integrity or inspect token claims. Stop if the VM, attached identities or operator path is no longer trusted; this is a benign lab, not a compromised-host evidence collector.

## 5. Change, compare, watch and restore

Record D1 with the exact rule scope, approver, risk of delay, peer-health bound and next UTC review time. Then:

```powershell
.\lab.ps1 -Action Contain @common -Approved
.\lab.ps1 -Action Probe -Phase contained @common -Approved
```

The runner uses a separate `containment.bicep` incremental deployment containing only the worker NSG, not a redeployment of VMs, identities, DNS or Storage. It first requests machine-readable ARM what-if (`--no-pretty-print`). It refuses reported changes outside the worker NSG; noisy/unresolved results require manual review, not a bypass. It also refuses an unexpected live worker rule inventory. Contain requires saved baseline 200 results from matching sources/identities within 15 minutes. That check cannot authenticate a local record or replace the operator's declared evidence budget.

Inspect the actual effective NIC NSG and route state through separately authorized read-only inspection. Match the `ContainStorageHttps` priority, protocol, direction, destination and peer NIC before drawing a conclusion. An applied configuration and a request outcome are different evidence classes. Optionally use documented Network Watcher IP flow verify for a recorded five-tuple if your sandbox already has a separately approved instance and permissions; the toolkit neither creates one nor claims its rule evaluation proves service behavior.

Stop on peer failure, unexpected success, identity failure, missing output, changed scope or lost control. Preserve evidence and create D2/ESCALATE rather than silently broadening the rule. A timeout alone remains inconclusive. For containment watch, record last evidence time, maximum gap, automation owner/disposition, temporary exceptions, expiry and next accepted handoff. Do **not** run competing reconcilers or intentionally create persistence.

Only after a new service-owner-approved lab recovery decision:

```powershell
.\lab.ps1 -Action Restore @common -Approved
.\lab.ps1 -Action Probe -Phase restored @common -Approved
```

Restore removes the synthetic worker deny in this disposable lab. It does not certify an old production baseline as safe, repair DNS/routes or rebuild identity. Do not remove emergency controls from a real incident using this example.

Create a **new** private decision record using the offline fixture as a field guide. Replace every fictional identity, owner, approval, scope and time; remove its synthetic observations; set its class to OBSERVED only for actual work and record the identity/recovery limits. Keep a versioned copy of each decision rather than overwriting D1.

```powershell
node .\validate.cjs .\private\actual-record.json --evidence $evidence
```

The CLI combines your record with the six exact phase files. Missing files, mixed classes and invalid comparisons fail explicitly. Actual Python timestamps and token-free outcomes are collected, not manufactured by the validator. For an ongoing watch, stale evidence or a reached decision deadline appears as a gap. After closure, preserve these as historical observations rather than claiming current health.

The runner sends the guest script as an LF-only, BOM-free `@file` to avoid Windows multiline quoting loss. It retains that generated shell script and each Run Command response in the private output directory for diagnosis. Those are restricted lab artifacts, not files to publish or package. The completed six-phase record and validator verdict remain useful even when the verdict is deliberately INCONCLUSIVE.

## 6. Cleanup and partial failure

```powershell
.\lab.ps1 -Action Cleanup @common -Approved
```

Cleanup checks the dedicated lab tag, successful deployment record and exact allowed top-level resource IDs before deleting the group. Unexpected resources cause refusal. It waits and verifies the group no longer exists. That deletes canary resources and synthetic identity grants with their scopes; retain only approved redacted evidence.

If deployment failed before outputs existed, or unexpected resources/what-if noise block automation, **the script deliberately cannot certify cleanup**. Inspect the exact tagged group in the approved subscription, identify every partial resource and seek explicit group-deletion approval. Use the Azure portal's resource-group deletion for that one confirmed lab group, verify its absence and account for any external resources. Do not delete another group or broadly disable Network Watcher/provider resources. Apply your retention policy to the lab key and local evidence separately.

## Coverage across the six articles

| Article | Runnable reader task | What remains outside this minimal lab |
|---|---|---|
| 1. Blast radius | Record exact source/principal/container and compare identity-specific operations | Exhaustive reachable-asset inventory and real incident triage |
| 2. Containment call | Own D1, rejected alternative, impact bound, deadline and escalation; invalidate malformed records | Whether a real service owner accepts the risk |
| 3. Keep containment intact | Recheck fresh operations; detect expired evidence/exception in validator; tabletop automation proposal | Implemented drift detector and surviving-session experiment |
| 4. Identity recovery | Demonstrate that a network change leaves identity authority in place; document untested recovery claim | Token revocation, federation repair and identity rebuild |
| 5. Restore safely | Authorize a separate lab restoration and rerun worker/peer operations | Production DNS/route recovery and independent recovery-channel assurance |
| 6. Rehearse | Run the six-inject pack; challenge missing evidence instead of inventing it | Actual workshop metrics until a reader runs and records a session |

Success is a reproducible, attributable **bounded comparison plus an honest decision record**, not a universal green badge. A contradiction or justified inconclusive conclusion can demonstrate the learning better than a fabricated pass.

## Package maintenance

From the repository's toolkit source folder, run `.\build-package.ps1` to regenerate the ZIP and sibling SHA-256 file. The builder copies only named public files and the two canonical worksheets, never a `private` folder, SSH key or collected evidence. Verify the archive's hash before sharing; verify the deployed bytes and download URL separately before announcing reader availability. A checksum detects mismatched bytes, not the trustworthiness of its source.

## Authoritative references and recheck

Reviewed 2026-10-26; recheck before deployment, especially before 2027 publication:

- [NSG statefulness and rule processing](https://learn.microsoft.com/en-us/azure/virtual-network/network-security-groups-overview)
- [Linux VM action Run Command and access requirements](https://learn.microsoft.com/en-us/azure/virtual-machines/linux/run-command)
- [Managed identity token use from a VM](https://learn.microsoft.com/en-us/entra/identity/managed-identities-azure-resources/how-to-use-vm-token)
- [Authorize Blob access with Azure RBAC](https://learn.microsoft.com/en-us/azure/storage/blobs/assign-azure-role-data-access)
- [Storage public network controls](https://learn.microsoft.com/en-us/azure/storage/common/storage-network-security)
- [Storage private endpoints](https://learn.microsoft.com/en-us/azure/storage/common/storage-private-endpoints)
- [Private endpoint DNS](https://learn.microsoft.com/en-us/azure/private-link/private-endpoint-dns)
- [IP flow verify: configuration evaluation and prerequisites](https://learn.microsoft.com/en-us/azure/network-watcher/ip-flow-verify-overview)
- [Azure pricing calculator](https://azure.microsoft.com/en-us/pricing/calculator/)