What the HOL scan checks, and what it cannot catch
Updated .
HOL scans the source of plugins, MCP servers and skills without running them. The scan finds known risky patterns in the files it reads. It misses encoded or assembled commands, payloads kept in other files or downloaded later, and instructions written as plain prose. A scan result is evidence about those files, not a guarantee about the software.
What the scan reads
The scan is static: it reads files and never runs the plugin, server or skill.
- Plugins in the catalog
- The manifest and publishing metadata, skill instruction files, MCP server definitions, and repository files that show maintenance: README, SECURITY.md, package manifest, lockfile and license.
- Skill instructions
- Shell commands that read secrets or send data to a remote host, including commands split over lines or hidden in HTML comments, and instructions that pipe a downloaded script into a shell.
- Servers from the official MCP registry
- The public GitHub repository the registry entry names, at the commit HOL fetched. Listings without a public GitHub source are not scanned.
- Skills from skills.sh
- The skill's own folder in its source repository, not the rest of the repository.
- Skills from ClawHub
- The files of one published version, downloaded from ClawHub. A newer version stays off the listing until it is scanned. Skills ClawHub hides or marks as suspicious or malware are not listed.
What it cannot catch
Static scanners for agent skills can be evaded. Published tests in 2026 bypassed every skill scanner they tried, using the same techniques listed here (CSA research note, obfuscation tests). The HOL scan misses:
- Encoded commands, such as base64 decoded into a shell, command names written as escape sequences, or names broken up with zero-width characters that a model may retype without them.
- Commands assembled at run time from variables or string pieces.
- Payloads kept in a bundled script or program that the instructions only call.
- Code downloaded after install. The scan can flag the download instruction but cannot see what arrives.
- Instructions written as plain prose, such as asking the agent to paste a secrets file into its reply.
- Behaviour that only appears when the software runs, and changes pushed after the scanned commit.
Evasion corpus results
Each case is a synthetic, inert skill that tries one technique. The open-source scanner runs on every case in CI, so these results change only when the scanner does. 11 evasion cases, one of them a plain control: 4 detected, 7 missed. The benign control is not flagged.
| Technique | Result |
|---|---|
| Plain command that reads .env and posts it (control) | detected |
| Same command split over continued lines | detected |
| Command inside an HTML comment renderers hide | detected |
| Downloaded script piped into a shell | detected |
| Zero-width spaces inside every command and file name | missed |
| Base64-encoded command decoded into a shell | missed |
| Command name written as hex escapes | missed |
| Command assembled from shell variables | missed |
| Command kept in a bundled shell script | missed |
| Bundled Python that runs a base64 payload | missed |
| Plain prose asking to copy .env into the reply | missed |
| Advice naming risky patterns (benign control) | not flagged |
How coverage is counted
- Coverage is reported as scanned out of listed, with the date it was read. Each source (catalog plugins, MCP registry servers, skills.sh skills, ClawHub skills) is counted separately.
- A listing counts as scanned only when its latest scan finished and returned a result. Pending, failed and skipped scans count as not scanned.
- MCP registry servers and skills.sh skills are counted against the listings with a public GitHub source, and the share of all listings is shown next to it.
- Scan order is decided by evidence and demand, never by payment or membership.
Rules for published numbers
- Counts from 1 to 4 are shown as "fewer than 5". A rate is shown only when both of its counts are shown.
- Windows are closed UTC days, 28 days by default and at most 92, so the same window always gives the same answer.
- Every number names its count and its window. A source that is unavailable is reported as unavailable, never as zero.
Why some numbers changed
- : Human traffic counts on plugin pages now use filter mkt-002.v2. It also removes sessions from a headless browser screen size, a Linux fleet with a fixed viewport, and a no-interaction proxy fleet. On October plugin pages these were about two thirds of the sessions the earlier filter counted as human, so human counts are lower from this date. Raw counts are still recorded, so both series can be compared.
- : Visitor country comes from the visitor address. Before this date, captures sent through the HOL proxy were located at the proxy, so earlier country splits are not used.
- : Human traffic counts now use filter mkt-002.v3. Checked against Google Search Console clicks for 28 days to 2026-10-27, the earlier filter removed some real Google visitors through its no-interaction timezone rule. That rule no longer applies to a session that arrived from a search engine, which adds about 1% to human counts. Raw counts are unchanged.