--- name: run-scenario description: Drive a reproducible request scenario against a locally running service - ad-hoc requests, or a stored k6 benchmark from .odd/benchmarks/ - and record it verbatim, so the telemetry it produces can be compared with a later run. Use when traffic must be generated before observing a service, when a stored k6 benchmark must be run, when an observation report needs a replayable scenario, or when verifying after a fix that the same scenario now measures better. --- # Run a Scenario One protocol on both ends of the ODD loop: the same commands that produced the numbers in an observation report produce the numbers that verify the fix. A scenario that cannot be replayed verbatim makes before/after comparison an impression, not a measurement. **A step this package ships a script for is run, never rewritten.** The ad-hoc drive is this skill's `scripts/drive_scenario.py` (step 2); a stored benchmark's replay is `k6-guides`' `scripts/replay_benchmark.py` (`references/benchmark-replay.md`). A curl loop, a helper with a `sleep` in it, a poller written for either yields a different command each time — two runs then measure two things. ## Read by situation Steps 1 to 5 and the rules are the method; a reference is read by the block that applies, never whole: | Situation | Reference | | --- | --- | | Every drive: the clean-base order and the reset decisions, the identity the queries are qualified by and the instance read from the rows, the run's t0 after the warmup — a port already served, a remote target the run cannot launch, a restart that is not possible, a reset that is forbidden | [references/run-identity.md](references/run-identity.md), the block that applies | | An iteration that is expensive or non-deterministic; the watch of a run someone else drives, on a backend that ships no watch script | [references/long-scenarios.md](references/long-scenarios.md) | | A stored k6 benchmark under `.odd/benchmarks//` — driven here, or driven elsewhere and only watched | [references/benchmark-replay.md](references/benchmark-replay.md), in place of step 2; its watching section for a run someone else drives | Start with the identity reference, then follow the steps below. ## 1. Decide what to exercise In order of preference: 1. **The caller's list** — endpoints, payloads, and counts given in the mission. Use them as-is; do not "improve" them. 2. **Traces already in the stack** — the operations the stack's traces name for this service are what the service actually serves: the service probe (the `setup-local-stack` skill's) and the backend reference's discovery script both list them — read that output rather than querying again. 3. **The service's own contract** — an OpenAPI/Swagger document, a route table, a CLI entry point in the repository (read-only). Prefer a handful of representative operations covered properly over every operation covered once. Note anything you deliberately left out. ## 2. Drive it What you decide: the operations (step 1), the count per operation (step 3), concurrency or not, the signal the flush wait is sized by (step 5), the run slug and the prompt's name. Everything else is the script's: ```bash python3 /run-scenario/scripts/drive_scenario.py http://127.0.0.1: --run-slug --prompt --op 'GET /api/users' --op 'POST /api/orders {"sku": "A1"}' --count 30 --wait-for traces --out / ``` The whole surface: the base URL (positional; `127.0.0.1`, never `localhost` — refused; the mission's or the `Machine:` line's when it carries one), `--run-slug`, `--prompt`, `--op 'METHOD PATH [COUNT] [BODY]'` (repeatable; the count overrides `--count` for that operation, the body goes out as `application/json`), `--ops ` (one per line), `--count` (default 30), `--warmup` (default 5, discarded), `--concurrency` (default 1), `--prefix` (default `0ddc0ffe`), `-H 'Name: value'` (repeatable; a credential-named header's value is never recorded), `--timeout` (per request, default 60 s), `--wait-for traces|metrics|none` (default `traces`), `--scenario ` (default the slug), `--out `, `--detach`, `--status [--wait ]`, `--dry-run` (the plan and the command, nothing sent), `--json` (the record as one object) — nothing else, so `--help` has nothing to add and the file nothing to read. It prints step 4's record, three lines of which are `` placeholders to fill, and writes `drive-record.json` and `requests.jsonl` under `--out` — the directory this run alone owns, `//` unless the caller named one: a sibling's file overwriting yours is silent. A scenario longer than a tool call is the same invocation with `--detach` (returns at once), then `--status --wait ` — exit 3 when the bound passes with the drive still going, run it again. Never a poller or a helper around it, never a turn ended to wait: as a subagent, ending the turn ends the mission. ## 3. Iterate enough to quote a number - **>= 30 requests per operation** before quoting a p95 — an operation being the unit the service serves distinctly: on an HTTP server the method and the route together, so two verbs sharing a route need the count each. Below that, report observations, not quantiles. - **~100** before quoting a p99. - Sequential by default. If concurrency is part of the question, state the level explicitly — it changes every latency number. - Keep inputs deterministic: fixed IDs, fixed payloads, a fixed seed. A random payload is not replayable; if randomness is unavoidable, record the seed. - An iteration that is expensive or non-deterministic: `references/long-scenarios.md`. ## 4. Record verbatim The record is what the script printed — the deliverable, quoted in the report and re-run from its `Commands:` line after a fix: ```text Scenario: Base URL: http://127.0.0.1: Listeners: none # or: :8000 served by 41234 uvicorn (127.0.0.1) and 51022 com.docker (*) Backend: Instance: Identity: User-Agent "odd-/" (+ "-warmup" on the warmup); traceparent "00---01", run8 = sha256()[:8] = , trace ids start ; one sequence run-wide from 1 Warmup: 5 requests per operation (discarded; seq 1-10) Load: 30 requests per operation, sequential (seq 11-70) Started (UTC): 2026-08-17T10:04:12Z Ended (UTC): 2026-08-17T10:05:03Z Query points: 1 (after Ended; after the 60 s flush wait for traces) Commands: python3 /run-scenario/scripts/drive_scenario.py http://127.0.0.1: --run-slug ... # as printed Requests: GET http://127.0.0.1:/api/users x30 (seq 11-40) -> 200:30 POST http://127.0.0.1:/api/orders {"sku": "A1"} x30 (seq 41-70) -> 201:28, error:2 Not reproducible: ``` - `Started`/`Ended` are the observation window of every query. - `Backend:` carries how the stack was (re)started **including any `env`**: a bare `odd_stack_reset` reapplies the env persisted in `stack_config.local`; only the credential-named variables the reset result lists under `env_not_persisted` must be passed again. A reset the mission requires (`references/run-identity.md`) is a `Commands:` line of its own, with its env and its reason. - `Query points:` is the one point the script pays for; a mission that must read the store at several points (one store per reset in a lifecycle test) runs the script once per point and lists them here, each with its reason. - A stored benchmark's record is `references/benchmark-replay.md`'s. ## 5. Wait for the flush — once per query point - `--wait-for` is sized by the slowest signal the point reads: **~60 s** for traces to become searchable on the local stack (a trace fetch by id may work before search does — cross-check a suspicious search result against a fetch), **~10 s** for metrics. - Paid **once per query point, after the last request that point reads** — never per query, never per request batch: a wait after every batch turns a 3-minute scenario into 4 minutes of sleep. - Inside the turn: the script blocks through it, foreground or `--status --wait`; a background notifier fires after a subagent's turn has ended. - A remote backend's wait is `observe-run`'s to size (its documented ingest latency, or a bounded proof query): `--wait-for none`, then that. ## Output The record above (or `references/benchmark-replay.md`'s for a stored benchmark), ready to paste into an observation report — the run record, and the replay instruction in the measurement protocol — and ready to re-run unchanged after a fix. ## Rules - Drive the service only; never change its code or configuration to make a scenario nicer. If a scenario cannot run as given, report why. - Replay after a fix with the **same** commands, counts, warmup, and concurrency — one changed variable invalidates the comparison. - Same machine, same data volume, same environment where possible; if something differed between the two runs, say so next to the numbers. - A failed or partial run is data: record the failures and their counts rather than retrying silently until the numbers look clean. - A driven request that calls a paid model spends money — every warmup and every iteration of it. Like a drive at a remote target, it is confirmed by the caller before the first such request goes out: the operation, the model when known, and the request count with the warmup included; the mission block saying the spend is accepted is that confirmation, a stored benchmark's manifest never is — it authorizes the load's shape, not the bill. No confirmation, no drive: leave the model-calling operations out, or stop, and say which in the record. An operation calls a paid model when the mission says so, when the stack's traces already show `gen_ai.*` spans under it, or when the service's own contract names a model; an operation you cannot rule either way is left out too, named in the record as unruled — a stop-and-report, never a question the drive waits on.