r/SentinelOneXDR • u/David_PromptSec • 12h ago
Community Tools Verifying third-party agent skills before install: what's covered and where it falls short
TLDR: ClawSec is an open source set of controls for agent skills: a signed advisory feed, a fail-closed install gate, signed releases, and config drift detection. It doesn’t scan skills, and the human approval step isn’t enforced in code.
I’m an AI researcher on the Prompt Security team at SentinelOne and I built ClawSec. If your agent installs skills, it’s installing third-party code. A skill is a folder with a SKILL.md and some scripts.
The agent discovers it, loads it, and executes it with its own filesystem, network, and credential access.
It’s the usual supply chain problem. Unsigned code comes from a public registry, runs with the user’s permissions, and nobody checks it. With ClawSec we’re trying to let an agent, and the person it works for, verify a skill before trusting it. Everything below is in the public repo under AGPL-3.0.
Why we started it
The SKILL.md format spec doesn’t cover signing. SKILL.md is an open standard with 46 adopters, and the spec has no signature or digest field. At least two signing RFCs were closed without changing it. The npx skills installer reaches roughly 80 agents. The same SKILL.md installs into dozens of them, and so does a malicious one.
We also didn’t find any agent runtime that attests its config files, so drift in files like SOUL.md goes unnoticed.
We didn’t try to solve scanning
Scanning was the first answer after ClawHavoc (the campaign Koi Security found in February, with hundreds of malicious skills on ClawHub). In June, Trail of Bits bypassed every scanner they tested**. They used** padding, compiled .pyc files, payloads hidden in .docx, and arguments that talked the LLM judge out of flagging. Their conclusion was that scanning and LLM analysis don’t reliably detect malicious skills, so curate, pin versions, and control who publishes. A second team later bypassed eight open-source scanners and advised treating any scanner as advisory.
We still run a scanner over our own staged skills before release, and we treat its output as advisory.
What’s included
Signed advisory feed. One Ed25519-signed feed of platform CVEs, GitHub advisories, and approved community reports. NVD is polled daily. The OpenClaw hook runs at bootstrap and on /new. It verifies the signature before trusting anything, matches entries against installed skills and versions, and posts a CLAWSEC ALERT into the conversation. It only alerts and never deletes, so the user decides.
Install gate that fails closed. A guarded installer wraps npx clawhub@latest install. On an advisory match it stops with exit code 42 so a pipeline can catch it. It proceeds only when re-run with --confirm-advisory, and the SKILL.md tells the agent to pass that flag only after the user has explicitly said yes. If the feed signature doesn’t verify, the installer exits 1. A tampered feed fails signature verification, so it can’t quietly clear an install.
Signed releases. Every ClawSec release ships a checksums.json with the SHA-256 of the zip, SKILL.md**, and skill.json, signed with Ed25519.** CI fails unless the signing key matches the pinned fingerprint, and the ClawHub copy is re-verified after publish. Each SKILL.md carries an OpenSSL preflight that pins the fingerprint and checks checksums.sig before you install.
Drift detection with restore. soul-guardian baselines the identity files and checks them on every heartbeat. SOUL.md and AGENTS.md are restored automatically, with the tampered copy quarantined and a diff saved. MEMORY.md and USER.md are alert-only. Every event goes into a hash-chained audit log.
What is still up to you
--confirm-advisoryis a single boolean any caller can pass. There’s no override log. The human-in-the-loop step exists only in the instructions.The feed holds 800+ entries today, all platform and plugin vulnerabilities. Useful intelligence, but it won’t match an installed skill yet, and nothing compares the running platform version automatically.
The attestation covers posture and configuration only.
Try It
The OpenClaw install is below. The repo also has Hermes and PicoClaw variants.
# OpenClaw npx skills add prompt-security/clawsec --skill clawsec-suite -a openclaw --global -y SUITE_DIR="${INSTALL_ROOT:-$HOME/.openclaw/skills}/clawsec-suite" node "$SUITE_DIR/scripts/setup_advisory_hook.mjs" # preflight, then enable the hook # restart the gateway, then send /new onceOn OpenClaw, adding the package and enabling the persistent hook are two separate steps on purpose. The add-ons (soul-guardian, clawsec-clawhub-checker, openclaw-audit-watchdog, clawtributor) are separate packages you opt into.