Why agent skills work — and why they stop working at scale

I've spent the last several months building skill libraries for the agents I run in production — Claude Code skills for blog-art, code-review, dataviz, a growing shelf of them for Auracle. So a paper that actually measures why skills help, instead of just reporting that they do, got my attention immediately. "Demystifying Agent Skills: Why They Work — Until They Don't" (Jiang, Huang, Xing, Wu, Gao, Cao, Wang, Liu, and Li, submitted 14 Aug 2026) runs controlled experiments across multiple benchmarks, agent harnesses, and LLMs, then normalizes 8,135 trial records and hand-codes 238 unique trajectory labels into a taxonomy of three categories and twelve skill-use modes. The result is the most granular account I've seen of what a "skill" — a structured, retrievable package of procedural or factual knowledge handed to an agent at inference time — is actually doing when it helps, and what breaks when it doesn't.

The headline: skills anchor behavior, not facts

The intuitive story about skills is that they inject knowledge the model doesn't have — an API's quirks, a library's correct invocation order, a domain fact. The paper's trajectory analysis says that's mostly wrong. Across the 238 coded cases, procedural anchoring accounts for 65.7% of skill uses, versus 4.5% for explicit knowledge injection. The dominant mechanism is an agent that already roughly knows what to do, but whose execution is noisy — it hesitates, retries, drifts into a slightly wrong tool call, or loses the thread over a long trajectory. A skill, in this reading, functions less like a fact sheet and more like a checklist taped to the wall: it doesn't tell the agent something new, it keeps the agent from wobbling off a path it was already capable of finding.

That reframing matters for how you'd build and evaluate a skill. If skills were primarily knowledge injection, the design problem is one of coverage — write down everything the agent might not know. If skills are primarily procedural anchors, the design problem is one of stability — write down the sequence of steps in a form the agent will follow faithfully under pressure, even when it's tired, distracted by a long context, or facing a slightly unfamiliar variant of the task. The paper backs this with a quantitative result too: skills improve over Workflow Memory (a competing inference-time technique) by 6.06 points in matched comparisons, which is consistent with anchoring effects being the more reliable lever than knowledge recall.

The bottleneck isn't the skill — it's finding it

The more consequential finding, especially for anyone scaling a skill library past a handful of entries, is about retrieval. The study isolates retrieval difficulty as a separate variable from skill quality, and the numbers are stark: actual-use precision falls from 29.6% at 5 skills to 3.3% at 100 skills. That's not a gentle degradation — it's a near order-of-magnitude collapse over a 20x growth in library size. The implication is that the thing most builders spend their effort on (writing better skills) is not the thing that fails first at scale. The thing that fails first is the agent's ability to pick the right skill out of a growing pile.

This is exactly the design shift Claude Code's own skill system has been circling — the using-superpowers pattern of a lightweight index plus keyword-triggered specificity — and this paper gives that instinct a number to defend it. It's also a reason to stay suspicious of any team's plan to just keep adding skills as the fix for agent reliability: past a certain library size, each additional skill is adding retrieval noise faster than it's adding capability.

The paper adds a wrinkle that keeps this from being a simple "retrieval is broken" story, though. Confusable distractors — near-duplicate or superficially similar skills — measurably hurt offline identification (an evaluator's ability to say "this is the right skill for this task"), but downstream task success stays comparatively stable. In other words, exact ground-truth invocation is neither sufficient nor necessary for the agent to succeed. Agents seem to route around imperfect retrieval more often than clean benchmark numbers would suggest, at least up to a point. That's a genuinely useful finding if you're building eval harnesses: measuring "did it retrieve the intended skill" is a different question from "did it complete the task," and the gap between them is wider than most retrieval-accuracy dashboards imply.

Where skills actually fail

The taxonomy's failure modes are also worth internalizing directly, because they're the ones I'd expect to keep recurring in production: brittle assumptions baked into a skill (steps written for one environment that silently break in a slightly different one), incompatible context (a skill invoked in a situation its author never considered), and insufficient adaptation (an agent that follows a skill's procedure too literally when the task actually needed a variant of it). None of these are retrieval failures — they happen after the right skill has already been found and loaded. That's the second axis of the paper's contribution: skills don't fail in one place, they fail at the boundary between "found the right procedure" and "correctly generalized it," and aggregate success-rate metrics collapse both failure modes into a single number that tells you nothing about which one you're looking at.

The practical takeaway

If you're building or maintaining a skill-based agent system, this paper argues for two concrete changes in where you spend effort. First, write skills as procedural scaffolding aimed at stabilizing execution under noise, not as knowledge dumps — the anchoring effect is where most of the value already lives. Second, treat retrieval precision as a first-class metric you track as your library grows, not an assumed constant — the 29.6%-to-3.3% collapse means a skill library that worked well at 5 or 10 entries can quietly become a liability at 50 or 100, with no change to any individual skill's quality. Aggregate task success rate will hide this until it's already hurting you.