Check your agent skills before merge with NVIDIA SkillEvaluator
SkillEvaluator scores an agent skill offline, predicts whether six agent harnesses will load it, and can measure the lift it gives a live agent.
TL;DR
SkillEvaluator is NVIDIA's multi-tier framework for agent skills and plugins. pip install skillevaluator gives you keyless commands that score a skill (quality-check) and predict whether Claude Code, Codex, Copilot CLI, Hermes, OpenClaw and OpenCode will load it (harness-check). Add a model key and Docker to run the same tasks with and without your skill and read the Skill Lift.
Why this repo matters
A skill can follow the Agent Skills specification and still never reach the model: the README's harness check shows Codex dropping a skill and Copilot CLI refusing one whose description is 4 UTF-16 units over its 1,024 limit. Beyond loading, you want to know whether the skill is well formed, safe, not redundant, and whether it actually changes agent behavior. SkillEvaluator splits that into three tiers (validation, deduplication, live agent evaluation) and ships a ci init command so every skill change is checked before merge. The repo is Apache 2.0 and labelled Experimental in its support policy.
Best practices
- Start keyless.
skillevaluator quality-check ./my-skillneeds no API key, scanner, GPU or Docker, and prints a score out of 100 with a grade (the README states the default pass threshold is 70). - Run the harness check before you ship. It is static, takes no model key, and its findings name the rule, what the harness does and the fix.
- Remember what static checks cannot say. The README notes
quality-checkis static only, so publication evidence staysINCOMPLETEuntil security and agent-impact checks run. - Know which tier gates. In
validate, Tier 1 gates, Tier 2 gates unless--no-block-on-dedup, and Tier 3 gates only with--block-on-agent-eval. - Review generated eval cases. The README says to review cases for Tier 3 datasets; templates omit unauthored negatives.
- Check readiness before paid runs.
skillevaluator doctor --env-mode dockerchecks readiness, and the README warns model and sandbox usage can incur charges.
How-to
Where do I start?
1. Install the CLI
Where do I start? You need Python 3.12 to 3.14, then:
pip install skillevaluator
The README also shows uv tool install --python 3.13 skillevaluator.
Next: Score a skill without a key
Try it
Make a throwaway my-skill/SKILL.md with a duplicated license: frontmatter key and a $ARGUMENTS placeholder in the body, then run skillevaluator harness-check ./my-skill -v. Fix what each harness reports and rerun until the not-loaded count drops to zero. (On our test run, Claude Code loaded it while Codex, Copilot CLI, OpenClaw and OpenCode did not.)
Pitfalls
- Scanners are required for full coverage. The README says
--no-llmdoes not remove scanner requirements; a scanner that cannot run isINCOMPLETEand exits nonzero. - Tier 3 needs real setup. Provider access, a supported agent and a running sandbox (Docker by default), and it can cost money.
- A passing quality score is not a publication verdict. Static quality alone leaves publication evidence
INCOMPLETE. - Experimental support. Community-supported, best-effort, no SLA.