Do skills actually help?
Each number comes from paired runs in a sandbox: the same task with the skill and without it, graded by deterministic checks and never by a model.
Effect by skill and model
Change in solved tasks, in percentage points. Below it, the 95% interval and how many times more tokens the skill used. 3 skills on 3 models.
| Skill | opus-5-5 | sonnet-5-5 | haiku-5-5 |
|---|---|---|---|
| test-driven-development6 tasks, 3 repetitions | +50+22…+786.6× | +100+100…+1008.6× | +44+11…+834.5× |
| receiving-code-review6 tasks, 3 repetitions | +110…+332.1× | +170…+502.9× | +170…+502.5× |
| brand-guidelines8 tasks, 3 repetitions | 00…01.1× | 00…01.4× | +290…+631.3× |
- Helps
- The whole interval is above zero.
- Not shown
- The interval touches zero. This is not proof of no effect.
- Harms
- The whole interval is below zero.
- No data
- Too few tasks or too many failed runs.
What the measurements show
- 1 of 3 measured skills have a proven gain on at least one model. None made results worse.
- 2 of 9 skill and model pairs did not need the skill: the model solved at least 95% of the tasks on its own.
- The typical skill costs 2.5× the tokens of working without it.
How well the scanner works
Rules only, on 289 malicious and 300 benign skills from MalSkillBench that were never used for tuning.
| Malicious skills flagged | 75.8% |
|---|---|
| Malicious skills marked as failing | 27.0% |
| Benign skills wrongly marked as failing | 1.0% |
| Benign skills with at least a warning | 16.0% |
About a quarter of malicious skills got no warning. A clean scan does not mean a skill is safe.
What this does not show
- A small task set cannot stand in for real work. A gain here is a gain on these tasks.
- Tasks marked as not reviewed were written by a model and have not been checked by a person.
- Models change. A result applies to the listed model and the exact skill version it ran on.
- Token multiples are not prices: providers charge different rates.
Read how we measure, or take the raw data as JSON.