Skillsmith

Deciding whether it should be a skill

First it checks whether the job needs a skill at all, or would work better as a rule in AGENTS.md, a hook or a one-off answer. If a skill makes sense, it lists what the repo already has, so the new one fills a gap instead of overlapping. The draft stays under 500 lines and has to pass a lint check before any test runs.

Testing it fairly

Both versions run fresh, with the same model, tools, task and scoring. I use one set of cases to improve the skill and a separate held-out set to decide whether to keep it. Once I've used a held-out result to change the skill, that case no longer counts as a fresh test. The judges get anonymous A/B packets in a balanced order, and anything with an exact answer is checked in code first. A critical failure counts against the skill even if its average score is higher.

When it doesn't pass

A new skill gets up to three serious attempts. If it still doesn't win on the held-out cases, it's archived. An existing skill keeps its last passing version until a revision beats it.

steps

5 · gate through keep or retire

skill body

under 500 lines · linted

revisions

3 serious attempts, then archive

gate versions

3 · older bundles still accepted

clients

Claude Code + Codex

model calls

0 · supplied by the host

runtime

Python standard library

status

MIT · a skill in github.com/ong6/skills