UX-OS

UX-OS / Method

How a skill gets built.

This is the method so far. It comes from building the information-architecture skill, the first one to go through full testing. Steps 1 to 5 are what I did on that skill. Steps 6 to 8 are what it showed I should add.

  1. Check what already exists

    Search the registry and the installed skills first. The information-architecture skill started because two public skills with the same name could not be installed together.

  2. Decide how to get it

    Adopt a good skill as it is, combine several when each covers part of the job, or write one when nothing fits.

  3. Check the licence

    Without an open licence I rewrite and never copy. Every source is credited at the bottom of SKILL.md.

  4. Write the skill and its tests together

    SKILL.md holds the method, references/ holds long material such as methodology, templates and a rubric, and evals/ holds realistic test cases.

  5. Test it against no skill at all

    Run each case with and without the skill using skill-creator, grade both, fix what failed and run it again. Record the score and the cost.

  6. Test the real name

    Ask a fresh session to quote the skill's entry in its list. With this many skills installed, most appear by name only, so the name has to say what the skill is for and rewriting the description changes nothing. Test with the real name, because the tuning tool in skill-creator uses a copy with a temporary name, which can lose to a competing skill for that reason alone.

  7. Check Claude can reach it

    Confirm no skillOverrides entry hides it and no installed skill with the same name competes with it.

  8. Register it

    Add a row to the registry and write down how it was built.

What testing showed on the first skill.

The first round scored 24 of 25 with the skill and 23 of 25 without, which is almost no benefit. After fixes and sharper checks, the second round scored 29 of 29 with the skill and 22 of 29 without. The third round, on the published version, scored 28 of 29 against 19 of 29. In those last two rounds the skill made runs 49% to 73% longer and used 19% to 26% more tokens, so the cost belongs in the record beside the score.

The skill, its test cases and every grade are on GitHub.

Trigger testing then showed how much the name matters. The tuning tool in skill-creator scored 0 of 10, with a rival skill winning 9, but its test copy carries a temporary name. With the real name, Claude chose the skill on 9 of 10 requests and on none of 10 near-misses.

When a skill changes.

Rerun its evals after any edit instead of changing it ad hoc.

Back to the sets