Only 16% of 'Docs' Commits Were Docs. Re-Tier Review.

Your change-review policy has a documentation exemption. On agent plugins, that exemption is where the behavior changes live. A study of 77,773 commits across the Claude Code plugin ecosystem, submitted to arXiv on 28 August, re-classified every commit by what its diff actually did rather than by what its message claimed. Of the 8,007 commits labelled docs, only 16% were documentation for humans. The remainder were feature and bug-fix work on the natural-language instruction files the model reads at inference time.

That is not a taxonomy quibble. If your organization runs the standard risk tiering — full review for code, light-touch or automated approval for docs — you have wired the low-risk path directly to the file type that determines what an autonomous agent does on a developer's machine. The commit message is the routing signal, and on this class of repository the routing signal is inverted.

What the Study Actually Measured

The paper is a maintenance study, not a security study, and its authors make no security claim at all — that reading is mine, and you should weigh it as such.

Ahmed Hereiz, Yingzhe Lyu, Hao Li, Bram Adams and Ahmed E. Hassan mined 1,926 repositories hosting 2,018 marketplaces, 8,351 plugins and 3,948 unique authors, covering the period from the October 2025 launch of the plugin system through the end of March 2026. It is the first empirical measurement of this ecosystem, and the ecosystem is not small or static: plugin-touching commit activity grew 8.8× in six months, from 2,923 commits in October 2025 to 25,618 in March 2026.

Their classification runs in two stages. Regular expressions matched the 51,141 commits (65.7%) that already carried a Conventional Commits prefix such as feat: or fix:; a model inferred the rest. Then two raters manually re-coded a stratified random sample of 700 commits — 95% confidence, 5% margin, minimum 50 per commit type — against what the diff did. Inter-rater agreement was κ=0.671. Those are the numbers to hold onto, because the per-type percentages below come from strata of 50 commits each, not from the full corpus.

Four Commit Labels That Mean the Opposite Here

Four conventional commit types describe a different activity in agent plugins than they do in ordinary software, and all four point the same direction.

An instruction file is a Markdown document — a SKILL.md, an agent definition, a command — that the agent loads into its context and follows as direction. It is not documentation about the system. It is the system, in the same sense that a shell script is.

Label	What it means in agent plugins	Share of sample
docs	Modifies instruction files read at inference time	74% (n=50)
refactor	AI-facing Markdown restructured "without intended behavior change"	50% (n=50)
perf	Model tier assignment, orchestration, hook tuning, routing	48% (n=50)
style	Wording, precision or formatting of AI-facing instructions	36% (n=50)

The docs finding extends to population scale, with one caveat worth stating plainly. The authors distilled the manual coding into per-type rubrics and had GPT-5-mini apply them to every commit by diff rather than by message. Across all 8,007 docs-labelled commits, 3,353 moved to fix, 3,047 to feat and 279 to chore; the docs share of the corpus fell from 10.3% to 1.7% — 80% reassigned to feature or fix work. So the 16% headline is a model's reading of the diffs, and the hand-coded stratum is the more conservative number: two humans reading 50 of those commits left 26% as genuine documentation. The magnitudes differ. The direction does not.

Read the refactor row twice. A refactor is a change the author believes is behavior-preserving. On a Python function, a test suite adjudicates that belief. On a Markdown file a language model interprets probabilistically, nothing does. Half of the refactor commits in this ecosystem are restructurings of AI-facing prose asserted to be semantically neutral, with no mechanism anywhere that checks the assertion.

The coupling data reinforces it: 78% of co-changes between a skill's instruction file and its scripts are functionally coupled, meaning the prose and the code have to move together or the plugin breaks. That is a maintenance dependency with no analogue in conventional software, and no linter that detects it.

Your Review Tiering Is Wired to the Wrong Signal

Go look at what your pipeline does with a Markdown-only pull request. Most enterprises have three mechanisms pointed at exactly that file type, and each of them was a good idea before agents existed.

The first is CI path filtering. GitHub Actions supports paths and paths-ignore filters so a workflow skips runs that touch no relevant files, and **.md is the canonical entry in every such list. The second is CODEOWNERS, where *.md routes to a docs group rather than to the engineers who own the runtime. The third is auto-approval or auto-merge for changes classified as low risk.

There is a subtlety worth knowing here, because it cuts both ways. A workflow skipped by paths-ignore reports nothing at all, so a required status check stays pending and blocks the merge — the safe failure. A job skipped by a if: condition is reported as successful, which satisfies branch protection. Teams hit the first behavior, find it annoying, and migrate to the second. The migration converts a merge-blocking gap into a silently-passing one, and the file type it does that for is the file type this study says carries the behavior.

None of those three mechanisms is wrong for a README. All three are wrong for a SKILL.md. The distinction your tooling needs is not code-versus-prose; it is does the agent read this at runtime.

This is the same failure mode as auto-approving a third of pull requests while agents make those pull requests larger, and a cousin of letting one vendor's model both write and review the change. The control was calibrated against a population that no longer exists.

---

The Update Path Has No Review Surface

Installing a plugin is reviewable. Updating one is not, and this is where the scout's obvious fix — pin the SHA — does not hold up.

Anthropic's own documentation is blunt about the trust model: plugins and marketplaces are "highly trusted components that can execute arbitrary code on your machine with your user privileges", and the install warning states that Anthropic "doesn't control what MCP servers, files, or other software are included in plugins and can't verify that they work as intended." Plugins ship hooks that execute shell commands on lifecycle events such as PostToolUse, and a plugin may carry a top-level bin/ directory whose contents are added to the Bash tool's PATH while it is enabled.

At install time you get a real review surface. The /plugin detail pane shows a context-cost estimate, a last-updated date, and a Will install section enumerating the commands, agents, skills, hooks, MCP servers and LSP servers the plugin adds. On update, you get a notification telling you to run /reload-plugins. There is no diff.

Three mechanics compound that:

1. Updates land in the background. Claude Code checks for marketplace and plugin updates after session start with a random delay of up to ten minutes, then loads the new versions on reload or next launch. Auto-update is on by default for the official Anthropic marketplaces and off by default for third-party and local ones — and an administrator can turn it on organization-wide by setting "autoUpdate": true on an extraKnownMarketplaces entry.
2. The community catalog's SHA pin advances by itself. Plugins approved into anthropics/claude-plugins-community are pinned to a specific commit SHA — and CI bumps the pin automatically as the author pushes new commits. The pin is a tracking pointer, not a review gate. Pinning is only a control when you own the marketplace entry and set the ref.
3. Version resolution can strand you on either side. The plugin.json version beats the marketplace entry's, and both beat the git commit SHA. Declare a version and forget to bump it, and users keep a cached copy forever while the instruction files churn. Declare none, and every commit to a Markdown file becomes a new version that propagates on refresh. Anthropic's docs warn against setting version in both places at all, because a stale manifest version silently masks the one set in the marketplace entry.

The precedent is not hypothetical. In December 2024, Cyberhaven's Chrome extension was compromised via a phished publishing token, and the malicious build reached users through the ordinary auto-update path with no user interaction, over a Christmas window when nobody was watching; the same campaign hit more than 30 other extensions. Every one of those extensions was on somebody's allowlist. The allowlist was a decision about an identity, and the thing that shipped was a version.

Instruction files raise the stakes further, because prose is the injection surface. We have already seen agents hijacked through text they were told to read, and agent configuration used as malware persistence after the packages were pulled. A sentence added to a SKILL.md needs no exploit.

The Case for Doing Nothing

The strongest counter-argument deserves stating properly, because it is partly right.

Plugin blast radius is a developer laptop, not production. The study found that 72.1% of marketplaces host a single plugin — these are overwhelmingly individuals publishing their own tooling, not a package registry with transitive dependency graphs. Only 6.0% of repositories are single-commit dumps, so maintainers are largely present and engaged. And an organization that clamps down too early forfeits real productivity: 61.3% of plugins target software engineering tasks, with code generation at 25.9% and infrastructure at 16.6%.

Fair. The proportionate response is not to ban plugins. It is to stop pretending a Markdown diff is a documentation diff, and to notice that three community aggregators already account for 12.1% of all plugins in the ecosystem — concentration that looks a lot more like a registry than like 8,351 hobby projects.

What To Do

This Week:

1. Grep your CI configuration for paths-ignore, paths, and *.md in CODEOWNERS. For each hit, ask whether the excluded set can contain a file an agent reads at runtime — SKILL.md, agents/*.md, commands/*.md, CLAUDE.md, hooks.json. Move those paths out of the docs tier and into the code tier. This is a routing change, not a policy debate, and one engineer can finish it in an afternoon.
2. Inventory what is actually installed. claude plugin list shows what is present, and claude plugin details enumerates the commands, skills, agents, hooks, MCP servers and LSP servers each plugin contributes. You are looking for the hooks and the bin/ directories first.
3. Decide the auto-update posture deliberately rather than inheriting it. Third-party marketplaces default to auto-update off; if someone turned it on, or set "autoUpdate": true in extraKnownMarketplaces, know that and own it.

This Month:

4. Set strictKnownMarketplaces in managed settings, which is the one scope users cannot override — it sits above command line, project and user settings in the precedence order. An empty array is total lockdown, including the official marketplace; a hostPattern entry matching your GitHub Enterprise host is usually the right shape. Pair it with disableSideloadFlags, or --plugin-dir and --plugin-url route straight around the allowlist. Set disableCommandPluginSources too: the allowlist matches the marketplace a plugin comes from, not the entries inside it, so a command-source plugin inside an approved marketplace still installs — and it updates on its own cadence, re-resolving once per session and installing a new version whenever its hash changes, independent of the marketplace's auto-update setting and of DISABLE_AUTOUPDATER. That is the update path in this article with every brake removed, and it is the one an allowlist alone does not reach. This is the same lesson as Claude Code's move to auto-mode: prompts are not policy, managed settings are.
5. Mirror rather than depend. Fork the plugins you approve into a marketplace repository you control, set an explicit ref per entry, and review upstream changes into it on your schedule. Do not treat the community catalog's SHA as a pin — its CI advances it for you.
6. Add one line to your review checklist for AI-facing Markdown: what does this change instruct the agent to do that it did not do before? Ask it of refactor and style commits specifically, because those are the two labels that assert no behavior change and, half the time here, are wrong.

Before You Widen the Rollout:

7. Run the version-drift check. For every approved plugin, compare the declared version against the commit history of its instruction files. A version string that has not moved in three months while the Markdown has moved twenty times means your users are on a stale cached copy — or that the author is shipping behavior under an unchanged version number. Both are findings.
8. Get the plugin question into your agent governance review alongside model provenance and skills verification. One reviewer should be able to say what an approved plugin instructs Claude to do, from the file, without running it.

The Bottom Line

Every previous extensibility cycle taught this lesson and every one of them taught it late. Browser extensions, npm packages, VS Code marketplaces, mobile app stores — in each, the industry got install-time review roughly right and update-time review badly wrong, and in each the attack that mattered arrived as a routine version bump to something already approved. Agent plugins compress the cycle further, because the payload does not have to be code. It can be a paragraph.

The 77,773 commits are not the alarming part. The alarming part is that 8,007 of them said docs, and the tooling believed them.

Continue Reading

• Zalando Auto-Approves a Third of PRs. Agents Made Them Bigger.
• Same Vendor Wrote and Reviewed 208,145 PRs. Split Them.
• Claude Code Stops Asking Aug 14. Prompts Aren't Policy.
• npm Pulled the Packages. Your Agent Config Reinfects You.
• Agentjacking: AI Agents Hijacked via Fake Bug Reports
• NVIDIA SkillSpector: The $670K Agent Governance Gap, Fixed
• IBM Wired 6 Agents With A2A. Then Couldn't Upgrade Them.