Benchmark Methodology

Scenario design

Each scenario defines a fixture (fake file content or action) and a risk class. The risk class determines which security control should apply. Scenarios cover eight risk classes:

• Secret access \u2014 reading .env, .npmrc, or SSH keys
• Risky shell \u2014 executing potentially dangerous commands
• MCP server changes \u2014 registering or modifying MCP servers
• Tool poisoning \u2014 injecting malicious tool definitions
• Package risk \u2014 installing untrusted packages
• Safe control \u2014 running known-safe actions that should never be blocked
• Approval control \u2014 actions that proceed after user approval
• Receipt creation \u2014 generating audit receipts

The benchmark includes 11 scenarios across these eight classes.

Comparators

Each scenario runs against 4 comparators:

• Native default (native_default) — Harness default/native control with no additional security layer
• Native strict (native_strict) — Harness strictest documented control setting
• Guard default (guard_default) — HOL Guard default policy
• Guard strict (guard_strict) — HOL Guard strict policy

Harnesses

The benchmark covers 5 AI coding agent harnesses:

• Codex CLI
• Claude Code
• Cursor
• Gemini CLI
• OpenCode

Metrics

The benchmark records 13 metrics per result:

• Prevented before execution (prevented) — Number of risky actions blocked before execution
• Prompted for approval (prompted) — Number of actions that prompted for user approval
• Allowed (allowed) — Number of actions allowed without intervention
• Detection only (detection_only) — Number of actions detected but not blocked
• Unsupported (unsupported) — Number of scenarios not supported by this comparator
• False positive on safe action (false_positive) — Number of safe actions incorrectly blocked
• Median added latency (median_latency_ms) — Median latency added by the security layer
• p95 added latency (p95_latency_ms) — 95th percentile latency added by the security layer
• Setup steps (setup_steps) — Number of commands or steps required for setup
• Install/setup elapsed time (setup_elapsed_s) — Time to install and configure the security layer
• Network required (network_required) — Whether the security layer requires network access
• Data leaving machine (data_leaving_machine) — Whether data leaves the developer machine
• Receipt/audit evidence created (receipt_created) — Whether a cryptographic receipt is created

Outcome determination

The runner determines one of five outcomes for each scenario-comparator pair:

• Allowed \u2014 the action proceeds without intervention
• Prevented \u2014 the action is blocked before execution
• Prompted \u2014 the user is asked to approve before execution
• Unsupported \u2014 the comparator does not support this scenario type
• Failed \u2014 the comparator encountered an error

Outcomes are deterministic: running the benchmark with the same fixture data always produces the same results. No randomization or timing-dependent behavior is involved.

Reproducibility

The benchmark is fully reproducible. Run it locally with:

pnpm run guard:benchmark -- --fixture --json

The output JSON matches the dataset published on the benchmark page.