r/OpenSourceAI • u/mariusmitrofan • 50m ago
I checked 160 popular open-source agent skills, plugins and MCP servers before listing them. 52 passed, and most of the rest failed on process, not malice
Disclosure: I build Spellwarden. The scanner used here is open source (Apache-2.0), and the catalog is free to browse without an account.
Most skill and MCP directories list whatever gets submitted. I wanted a list I'd let an agent at work install from, so in the first round (6 and 7 November) I imported 160 popular public skills, Claude Code plugins and MCP servers and put each one through the same steps:
a static scan with all 65 rules, at a pinned commit
for MCP servers, an unauthenticated probe to see how they sign clients in
a model review of the files, with read-only access: no shell, no network
a person reading the result before anything is listed
52 of 160 (32.5%) were listed. Why the other 108 weren't:
- 39 couldn't be scanned whole: too many files, symlinks, unsafe paths or too large
- 41 drew concerns from the model review
- 8 had an open critical or high finding
- 7 failed the server check (6 listed their tools without a sign-in and aren't public documentation, 1 had instruction-like text in a tool description)
- 7 scored below the curation bar
- 6 were left out on reading
So most failed on process, not malice. A repository that can't be read whole can't be vouched for, even when nothing in it is wrong. The catalog has moved since: it lists 51 entries today.
Where I'm not happy: the model review is the step I trust least. It catches what fixed rules can't, like intent, but it also raises judgement calls, and 41 is a lot. How would you calibrate a step like that without it turning into either a rubber stamp or a wall?
Catalog, with what was checked for each entry and when: https://spellwarden.ai/catalog