A Claude skill that refuses to audit a screen it hasn't seen
Open source, with the eval that caught the version that didn't
Context
Interface work for clients produces two documents: an audit of what is broken, and a spec for what to build. Ask a general-purpose model for either and you get prose that cannot be wrong — improve the visual hierarchy, take care of the feedback. It reads well and it dies in the first meeting where somebody opens the actual screen.
I had thirteen HCI principles written down from years of doing this by hand. The question was whether they could be turned into something an agent executes rather than paraphrases.
Constraint
The interesting failure here is not technical: it is that the model is too willing. Asked to audit a dashboard with no screenshot, no access and no file, it does not decline — it writes a plausible audit. A findings section, an observed-evidence block, a severity rating, all about a screen it never saw. That output is worse than nothing, because it is indistinguishable from work until somebody checks.
The second constraint was direction. Criteria that evaluate an artifact and criteria that constrain a design are usually written as two separate bodies of theory. If they diverge, the audit and the spec stop agreeing with each other.
What I built
Thirteen principles, each with a verification question, observable criteria at published thresholds (contrast ratios, touch target sizes, response-time bands), typical violations and intervention levers. The same criterion read against something that exists is a test; read before it exists it is a requirement — one grid, two directions, and the skill has to declare which one it is working in.
Then the part that matters: an outcome for not assessable with the material provided, kept separate from not relevant, and a path that produces a detection plan instead of an audit when there is no artifact. Every hypothesis in that plan carries the line that would disprove it. Plus severity ranked by impact times frequency, a stated hierarchy for resolving conflicts between principles, and two deliverable templates.
Three evals, twenty binary assertions, run against two versions of the skill.
What happened
The first version failed exactly where I expected it to be strongest. Given “our dashboard sucks, 40 identical widgets, nobody uses it, where do I start?”, it titled the document Usability audit, opened a Findings section, and wrote an Observed evidence block about forty widgets it had never seen, at severity High. It asked for ninety days of analytics and five to eight shadowing sessions per role. It never asked for the screenshots. Four of seven assertions passed.
The corrected version titles the same document Detection plan and writes “before the detection, an outcome does not exist”. Seven hypotheses, each with its disproof condition. Five specific requests, screenshots first, because they are the cheapest thing that unblocks the most. Seven of seven.
Across all three evals: 17/20 to 20/20, with the entire delta on that one case.
What the number does not say. One run per eval per version — enough to show a behaviour changed, not enough to call it a measurement. The twelve-second difference in run time is noise. And the grader found three internal counting errors in the corrected outputs — a summary saying “three open outcomes” with five in the table beneath it — that no assertion catches. I added a rule to the skill about recounting from the table; that rule is the one change this round of tests does not cover.
The skill, the evals, the grading and all twelve produced documents are in the repository. The full write-up is at /the-ux-decode/.