# Lab execution and evidence plan

**Preview status:** one authorized finite Azure trial is documented in VALIDATION.md, including an initial failed revocation comparison and a recorded readiness retry. Expected outcomes below remain hypotheses for every new run. Implementation and offline tests do not prove them.

## Preflight

Complete APPROVAL.md. Retain the filled approval outside the repo. Review source, compile Bicep and run offline tests. Record tool versions, dated pricing estimate, selected region/SKU/image, exact permissions, request/observation budgets and authorized cleanup deadline. Verify trusted operator recovery access before changes.

The proposed topology is `10.78.0.0/16`, with workloads in `10.78.1.0/24` and Blob private endpoints at `10.78.2.4` and `.5`. There is no peering, inbound VM connectivity or public Storage data access. NAT plus the broad AzureCloud HTTPS rule supports management; this **does not** prevent all exfiltration. Python is expected on the pinned Ubuntu image; missing runtime blocks seeding/probing and needs a reviewed source revision, not an unrestricted package-download workaround.

Stop on wrong subscription, unexpected resource or assignment, sensitive data, public VM exposure, external routes, unexplained rule changes, lost management, forecast cost breach, or an expired approval. Do not "fix" a failed test by relaxing global policy.

## Reader protocol

Use the same `--approval C:\private\approval.json --approved` arguments for each cloud command below. File paths are examples, not real approval values.

| Step | Command prefix | Required evidence before proceeding |
|---|---|---|
| 1 | `python lab.py deploy` | New tagged group, validated/what-if new-resource scope, successful deployment, manifest |
| 2 | `python lab.py seed` | R/S/C synthetic marker creation or exact existing-content verification; seeder grants removed and VM deallocated |
| 3 | `python lab.py probe --state baseline` | R/S/C marker matches; M returns BlobNotFound |
| 4 | `python lab.py state --state identity` | Only required-container worker reader grant; no network change |
| 5 | `python lab.py probe --state identity` | R matches; S/C authorization denied; M not found |
| 6 | `python lab.py state --state baseline` then `probe --state baseline` | Bounded lab-only restoration of broad reader grants and fresh baseline |
| 7 | `python lab.py state --state network` then `probe --state network` | R/S match; C times out under exact canary deny; M not found |
| 8 | `python lab.py state --state combined` then `probe --state combined` | R matches; S denied; C times out; M not found |
| 9 | `python lab.py state --state failure` then `probe --state failure` | Only required reader grant removed; R/S denied, C timeout; M also denied |
| 10 | `python lab.py recover` then `probe --state recovered` | Only required container grant restored; canary deny retained; R works and S/C remain prohibited |
| 11 | `python validate.py --evidence C:\private\evidence` | All six named comparison states present with three distinct attempts per case |
| 12 | Collect queries/management history, review, export and clean up | Telemetry, provenance, residual limits and actual deletion checked separately |

`state --state recovered` is also available on the normal failure path, but `recover` is the safer escape when the failure probe is inconclusive. It does not require successful failure observations and **never restores broad baseline grants**. It still refuses unexpected authority or altered NSGs. A partially applied transition stops; inspect retained evidence, then use narrow recovery if applicable or approved cleanup. Do not keep rerunning deployment: incremental deployments do not remove previously created role assignments.

## Bound the observations

Each probe is three fresh HTTPS connections per R/S/C/M case: twelve marker requests per state call. Each network request has a ten-second socket timeout; total operation/agent time is not a guaranteed wall-clock deadline. IMDS supplies an in-memory Storage token. The custom client first checks the workload's DNS answers, then pins the expected private IP with normal-hostname TLS validation. This is intentionally a controlled client, not proof of every production application's resolver/cache/session behavior.

Attempts have individual request IDs and UTC times. Redirects are not followed. Content is capped at 1,025 bytes and compared to the exact synthetic marker. The missing-object control M must not be confused with an authorization denial. A failed batch is retained in `attempts/`; convenience `state-case.json` files are the latest copies for validation. Never delete earlier failed observations.

Do not silently retry until a claim passes. Before the live run, set a maximum readiness window and at most two explicitly recorded readiness retries for role propagation, with an owner. If it expires, preserve failure/inconclusive status and stop the experiment. No claim of immediate token revocation follows from a fresh read.

Probe reports include actual source/principal, input hashes, expected target aliases, peer, status, marker comparison and error class. An authorization denial needs service evidence. A C timeout alone is **not causal proof**: review effective source rules/routes, stable inputs, R positive control, baseline/rollback evidence and timestamps. Combined-state C cannot prove authorization separately; use identity-only C.

## Telemetry and evidence review

The runner captures per-state configured worker NSG/role snapshots, NIC effective routes/NSGs, CLI failures, raw guest responses and parsed attempts. It does not certify DNS continuity, packet traversal, complete effective authorization, real-time drift detection or log coverage.

Use `queries/blob-request.kql` with the exact request ID, resource ID and URI. In Azure Monitor, confirm the table exists and the positive R read appears. Challenge it with a different request ID, same basename in another container and a nonexistent ID. Verify service status, requester object and timestamps; row count is not successful read count.

Retrieve management history for only the named lab group and bounded run interval, for example with `az monitor activity-log list --subscription <approved-id> --resource-group <approved-group> --start-time <UTC> --end-time <UTC>`. Store raw output privately with command, window and collection time. Validate caller/action/resource/status against the intended change; Activity Log correlations are not automatically Blob client request IDs. This read-only collection is manual so the reader must choose the actual run window. Record missing/late events as gaps; no automatic join or detector is claimed.

Review the source-to-question coverage in WORKSHEET.md. Event time, collection time, configuration acknowledgment and first observed effect are different clocks. No precise latency is inferred from a polling interval or late query.

## Recovery and teardown

`recover` changes only lab reader assignments to the required container and leaves the reviewed canary NSG deny. It is not a solution for a hostile administrator or a broken Azure control plane. If its preconditions fail, stop and use the named recovery owner; never open public SSH or remove every deny.

After evidence is exported and verified, run:

```powershell
python lab.py cleanup --approval C:\private\approval.json --approved
```

Cleanup verifies ownership/approval tags and the resource inventory, then deletes only the exact disposable group and confirms absence. It stays available after expiry with the original approval. A failed deployment leaves its tagged group for inspection/cleanup; unexpected inventory blocks automated deletion. Review any external/higher-scope assignments and billing lag separately. No zero-cost claim follows from group deletion.

## Release criteria

Require implemented/pinned source, a fresh authorized run, expected and actual outcomes with failures preserved, exact query controls, narrow recovery, confirmed deletion, attributable estimate/actual cost with uncertainty, sanitized evidence and independent reader replay. Automated schema/threshold passes alone are insufficient. Until that work is complete, publish neither "proved in Azure" claims nor invented before/after numbers.