Akula

How we check skills

What each metric means and what it does not guarantee.

Security

The scanner never runs a skill. It reads every text file and works in layers:

  • Code extraction. Code blocks inside documents are analysed like scripts, since malicious skills often hide code there.
  • Data flow. Python and JavaScript are parsed into syntax trees. Akula follows values through variables and functions, from secrets to network calls and from downloads or decryption to execution. Shell scripts get a line-level version of this.
  • Whole-package behaviour. Capabilities are combined across files and compared with what the description promises.
  • Dependencies. Manifests and install commands are checked for typosquats, install hooks and unpinned sources, then against OSV.dev, including its list of known malicious packages.
  • Instructions. Prose is checked for fake prerequisites, requests to move secrets, stealth, and false authority.
  • Patterns. Hidden Unicode, decoded base64, secrets, risky URLs, persistence and destructive commands. Split strings are joined back together first.
  • Destruction. Deleting broad or computed paths, and deletion the description never mentions.
  • Risk score. A statistical summary of the signals. It can quiet weak noise but can never raise a status on its own, because that falsely accused plain skills in our tests.
  • Test runs. On request, code runs in a sandbox among fake secrets, and we record what it reaches for. Run on demand, not on every submission.
  • Skill pairs. Skills that look harmless alone are compared for shared locations that would complete an attack together.

If a file cannot be read or analysed, that appears in the report: an unread file can never produce a clean result.

Measured detection

We test the scanner on public labelled data. On a fresh sample of the MalSkillBench corpus that was not used for tuning (289 malicious and 300 benign skills, rules only, no model review):

  • 75.8% of malicious skills were flagged, 27.0% as critical failures.
  • 1.0% of benign skills were wrongly marked as failing, and 16.0% got at least a warning.

In a separate test, 6 simple evasion tricks were applied to detected malicious skills. At least 98% stayed flagged after each trick, so a small share can slip through, and advanced evasion has not been tested.

About a quarter of malicious skills in that sample got no warning. The main gap is subtle prompt injection written as ordinary prose, which patterns and code analysis cannot judge. A reviewer stage with several models is built for this class, but it is not active yet and its accuracy has not been measured. So "No findings" does not mean a skill is safe, and a warning does not mean it is malicious.

Measured effect

The agent solves the same tasks in two modes: with the skill and without it. The skill is attached the way real agents do it: only the name and description are in context, and the agent opens the full text itself. This also shows whether the skill gets activated.

Success is decided only by deterministic checks: files, regular expressions, commands. A model never judges the result, because self-assessment inflates the numbers.

The interval is computed by bootstrapping over tasks. "Helps" appears only if the whole 95% interval is above zero. Fewer than 5 tasks, fewer than 2 repetitions, or more than 20% failures give the conclusion "Not enough data".

We also measure unrelated tasks where the skill is not needed: it should not get in the way of ordinary work.

Limitation. Generated tasks may not reflect real work, so they are flagged in the report and on their own never earn the "Use" recommendation. Some tasks were drafted by an AI assistant that had read the skill, and they carry the same flag until a person reviews them. The generator sees only the skill's description, not its text, so the tasks do not repeat its techniques.

Isolation

Skill code runs only in a sandbox with no network and no access to keys or host files. There is no mode without a sandbox.

Recommendation

  • Do not use: a critical finding, or harm on every measured model. A good effect does not cancel a critical finding.
  • Use with caution: there are findings, the benefit is not shown, the skill harms some models, or the tasks have not been reviewed by a person.
  • Use: no findings, benefit shown, run completed, tasks reviewed. This is not a guarantee of safety.
  • Effect not measured: no serious findings, but no measurements either.

Model review of the text only explains the report and does not affect the recommendation: the text under review can address the reviewing model itself.

Versions

Each report is tied to a hash of all the files in the bundle. If even one file changes, it is a new version and earlier measurements do not apply to it.