Skill Eval Pack
2026 · open source · claude code + codex
A useful-sounding skill is easy to write and hard to prove. I built this pack to compare the same agent on the same task with and without the skill, then keep it only when fresh heldout cases clear the gate. Code on GitHub.
Figure 01 · AI toolchain · three open-source projects
Write the instruction, prove it helps, check the facts
Skillpack supplies the instruction. Skill Eval Pack compares it with the same agent and task without the skill. Groundplane then checks declared output fields against recorded tool facts at runtime.
- Inline
- Queued
The control is the same agent without the skill
The baseline and treatment run in separate fresh contexts with the same model, tools, task and rubric. Development cases are for fixing the candidate. Fresh heldouts make the decision. Once a heldout result influences a revision, that case is retired.
The judge does not get the answer key
eval_gate.py prepare counterbalances anonymous A/B packets. Independent judges see complete transcripts and outcomes, while exact properties run through deterministic graders first. The decision unions critical failures instead of averaging them away. A treatment can score well overall and still fail for one core regression.
A failed revision does not replace the proven one
New candidates get up to three serious attempts by default. An existing skill keeps its last proven version until a revision passes. The lifecycle helper checks the attempt history, retired heldouts and final action, while the payload helper confirms that Claude Code and Codex loaded the exact frozen files.
- unit tests
- 54
- gate versions
- 3 · older bundles still accepted
- decision scripts
- 3 · evaluation, lifecycle, payload
- clients
- Claude Code + Codex
- model calls
- 0 · supplied by the host
- runtime
- Python standard library
- status
- MIT · github.com/ong6/skill-eval-pack
// AI TOOLCHAIN
Instructions, evidence, runtime facts
01 · instruct
Skillpack
The reusable instructions my agents load.
02 · prove
Skill Eval Pack
The blind test that decides whether a skill stays.
03 · check
Groundplane
The runtime boundary for declared facts.