Skillsmith

Deciding whether it should be a skill

Before anything is written, skillsmith asks whether the job needs a skill at all, or belongs in an AGENTS.md rule, a hook, a memory or a one-off answer. If a skill is right, inventory.py lists the skills and conventions the repo already has, so the new one fills a gap instead of overlapping one.

The draft starts from its trigger description, keeps the body under 500 lines, and passes lint_skill.py before any run. It installs as one skill for both clients: .claude/skills/skillsmith for Claude Code, exposed to Codex at .agents/skills/skillsmith.

Comparing with and without the skill

Both versions run in fresh contexts with the same model, tools, task and scoring criteria. I use development cases to improve the skill, then separate held-out cases to decide whether to keep it. Once I use a held-out result to make an edit, that case can no longer count as a fresh test.

Keeping the comparison blind

eval_gate.py prepare prepares anonymous A/B packets with the order balanced across comparisons. Independent judges see the full transcripts and results. Checks with exact answers run in code first. Any critical failure counts against the candidate, even if its average score is higher.

Keeping the previous version when an edit fails

New candidates get up to three serious attempts by default. An existing skill keeps its last passing version until a revision passes. A new skill without a clear heldout win whose uncertainty lower bound clears the gate is archived. lifecycle_gate.py checks the attempt history, retired heldouts and final action, while the payload helper confirms that Claude Code and Codex loaded the exact frozen files.

steps

5 · gate through keep or retire

skill body

under 500 lines · linted

revisions

3 serious attempts, then archive

gate versions

3 · older bundles still accepted

clients

Claude Code + Codex

model calls

0 · supplied by the host

runtime

Python standard library

status

MIT · github.com/ong6/skillsmith