HOL Guard Research / Methodology

How the AI Coding Agent Runtime Security Benchmark Works

This methodology defines 11 fake-data scenarios, assigns each a risk class, applies four documented comparator policies, normalizes each result into one of six outcomes, and exports the validated run as HTML, JSON, and CSV. Fixture mode never reads real secrets or executes real attacks. It is deterministic and reproducible, but it does not measure live harness behavior, production latency, or real-world false positives.

Scope of this report

This is a deterministic fixture model published by HOL Guard. It documents the export contract and policy rules; it does not claim independent validation or live harness performance.

01

The five-stage method

The runner keeps fixture creation, policy evaluation, normalization, and publication validation separate. That makes the current export deterministic and gives future live runs a clear place to add measured fields.

  1. 01

    Define fixture

    Create a deterministic fake file or action with no real secret or attack.

  2. 02

    Assign risk class

    Map the scenario to one of eight ordered taxonomy definitions.

  3. 03

    Evaluate comparator

    Apply one documented native or HOL Guard policy mode.

  4. 04

    Normalize outcome

    Record one of six typed outcome definitions.

  5. 05

    Validate and export

    Check publication fields, cardinality, and JSON or CSV output.

02

Scenario taxonomy

Each fixture receives one risk class. The ordered definitions below are the source of truth for taxonomy display and scenario counts.

  1. 01

    Secret access

    Attempts to read fixture credentials or private keys. 3 fixtures

  2. 02

    Risky shell

    Attempts to execute a potentially destructive shell action. 1 fixture

  3. 03

    MCP server change

    Registers or changes an MCP server command or URL. 2 fixtures

  4. 04

    Tool poisoning

    Encounters instructions embedded in a tool description. 1 fixture

  5. 05

    Package risk

    Installs a package-risk fixture. 1 fixture

  6. 06

    Safe control

    Runs a known-safe action that should remain available. 1 fixture

  7. 07

    Approval control

    Models an action that proceeds after explicit approval. 1 fixture

  8. 08

    Receipt creation

    Creates or verifies an action receipt. 1 fixture

03

Comparator policies

Every scenario is evaluated against the same four documented policy modes. The fixture runner currently uses one policy rule set for every eligible harness row.

  • Native default

    Harness default/native control with no additional security layer

  • Native strict

    Harness strictest documented control setting

  • Guard default

    HOL Guard default policy

  • Guard strict

    HOL Guard strict policy

The five harness rows establish coverage of the exported result shape. They are not five independently measured production implementations.

04

Six normalized outcomes

Each result is normalized to one of six states. The rubric distinguishes prevention, approval, detection, unsupported behavior, and safe-action interruption.

  • Allowed

    The modeled action proceeds without approval or prevention.

  • Prevented

    The modeled action is stopped before execution.

  • Approval required

    The modeled action waits for explicit user approval.

  • Detected only

    The modeled action is identified but not prevented.

  • Unsupported

    The comparator has no modeled behavior for the scenario.

  • False positive

    A known-safe action is incorrectly interrupted.

05

Recorded fields

Current fixture result fields

Scenario ID
Stable scenario reference used to join the fixture.
Harness ID
Eligible harness row represented in the exported shape.
Comparator
One of the four documented policy modes.
Outcome
One of the six normalized outcome definitions.
Median latency placeholder
Fixture-mode placeholder value, not a live latency measure.
p95 latency placeholder
Fixture-mode placeholder value, not a live latency measure.
Receipt-created flag
Whether the modeled fixture creates an approval receipt.
Notes
Additional fixture-mode context for the row.

Future live-run publication metrics

These metrics are reserved for measured live runs. They are not present on every current fixture result.

Setup steps
Live-run publication metric for installation and configuration work.
Setup elapsed time
Live-run publication metric for elapsed setup time.
Network required
Live-run publication metric for network dependency.
Data leaving machine
Live-run publication metric for data egress.

06

Reproduce the export

Run the fixture benchmark from the public source repository with either export flag. The result is validated before it is written, and the same run shape is available from the benchmark download endpoints.

JSON export

bun run benchmark -- --fixture --json

CSV export

bun run benchmark -- --fixture --csv

Open the benchmark report for the rendered 11-scenario matrix and direct JSON or CSV links.

07

How to interpret this run

This run shows

  • How the documented fixture rules map risky and safe actions to typed outcomes.
  • That the published export has complete five-harness coverage for every scenario/comparator tuple.
  • That the same deterministic input can be rendered as HTML, JSON, and CSV.

This run does not measure

  • Live harness behavior, exploit resistance, or production latency.
  • Real-world false-positive rates or network, container, and OS isolation.
  • Independent third-party validation of HOL Guard or native controls.

08

Limitations

  • Fixture outcomes do not measure real attack resistance.
  • Fixture latency values are placeholders.
  • Comparator behavior represents documented fixture rules.
  • Harness outcomes are identical because the current rules do not branch by harness.
  • Other harnesses may behave differently.
  • Network, container, and OS isolation are out of scope.
  • HOL Guard publishes this benchmark; it is not independent third-party validation.

Provenance

Document record

Author
HOL Guard Team
Technical reviewer
HOL Guard Team
Published
Reviewed / updated
Content version
1.1.0
Next scheduled review
Source repository
HOL Guard benchmark repository
Changelog
  1. 2026-08-02 · 1.1.0Clarified six outcomes, current versus future fields, five-stage method, and reproducible exports.
  2. 2026-07-26 · 1.0.0Initial publication alongside the fixture benchmark dataset.

Related report

Inspect the full fixture matrix

See every scenario, comparator mode, harness row, and downloadable result.

View benchmark