I launched WPL — the Wellness Plan Language — as a safety and governance layer for AI-generated fitness plans. The pitch was simple: a wellness plan is not a chatbot reply, and if it's wrong the client gets hurt, so it should be checked before a human ever sees it. I shipped a schema, a validator, a rule engine, and a benchmark that measures how often AI-built plans include exercises contraindicated for a client's injury or condition. The numbers looked good. Then I made myself do the uncomfortable thing: audit my own benchmark instead of marketing it. I found three ways the numbers were flattering me. This is what I fixed — even though the honest version is more modest — because a safety benchmark you can't trust is worse than no benchmark.

One. My filter had the answer key

The "rules a trainer would configure" in Lane B were derived from the same blacklist I graded against. So of course the structured plans scored near-perfect — I'd wired the answer into the filter. The benchmark wasn't measuring whether a sensibly-configured rule set catches a contraindication; it was measuring whether a rule set built from the contraindication catches the contraindication. That's circular, and it flatters the contract. I split them apart. Rules are now authored from the client's clinical picture — the injury, the condition, the constraints a clinician would actually state — independently from the grading key. The gap between "what a sensible trainer sets up" and "what the clinic actually requires" is now measured instead of assumed away. In one scenario, de-circularizing the rules flipped a near-zero violation count to seven. That seven is the honest number. It is not a regression in the contract; it is the contract being scored against rules that no longer contain the answer.

Two. The model graded itself

I'd been using each AI model to extract its own plan for scoring — the model wrote the prose, and the same model read the prose back into the structured list the scorer walks. The problem: a more capable model reads its own output more thoroughly and therefore surfaces more of its own violations, so it looks less safe. A confound that quietly punished the better models and rewarded the sloppier ones. I switched to one fixed, independent extractor for every trial. The same extractor model reads every plan from every model, with the same instructions. It is not a judge — it lists what was prescribed, it does not evaluate safety. The scoring stage no longer depends on which model happened to write the plan.

Three. I claimed determinism I didn't have, and my matcher failed open

The docs said "temperature 0, reproducible." In reality only three of my seven models even accept a temperature setting; the rest sample however they like. Claiming determinism I couldn't deliver is the kind of small dishonesty that erodes a benchmark's credibility one footnote at a time. Worse, the exercise-name matcher silently let push_ups slip past a push_up rule. A plural form walked straight through a singular rule — the matcher undercounted violations, which made every model look safer than it was. I fixed the matcher's plural handling, put confidence intervals on every rate, and rewrote the docs to say what the code actually does, not what I wished it did.

The engineering that makes the promise real

Under the honesty fixes, I also did the work that turns a benchmark claim into a deployable guarantee. Enforcement — the part that actually removes a contraindicated exercise from a plan — used to live only inside the eval harness. That's backwards: the thing measuring safety had the safety logic, and the thing shipping to clients didn't. I moved enforcement into the published validator. It now lives in @gymbile/wpl-validator@1.9.0 on npm (and wpl_validator 1.9.0 on Hex, with @gymbile/wpl-ai@2.1.0 alongside it). The same enforce() that the benchmark scores is the one production calls. There is no longer a gap between what I measure and what I ship. I made the compiler fail closed on safety paths. A typo in a safety section used to be a silent delete — the malformed rule just vanished and the plan compiled without it. Now it is a hard error. A safety constraint that can't be parsed stops the plan; it does not quietly disappear. I added a repairs ledger: every silent normalization the model made — every quiet fix the compiler applied to make a plan well-formed — is recorded, so a human reviewer can see exactly what was changed on the way to a served plan. Governance you can't inspect isn't governance. Then I wrote the test that pins the actual guarantee: a contraindicated exercise that a rule covers cannot survive compilation and enforcement. That's a deterministic property, checked on every build, not a hopeful average over a corpus. To make sure the move into the library didn't quietly break the frozen data, I ran a $0 census over the existing trials: 0 of 269 frozen Lane B trials fail-closed under the 2.x library. The frozen numbers are stable; nothing regressed in the move.

The honest coda

Here is the part that matters most, and the part a marketing post would never include: I am not publishing a new headline safety percentage. I have not re-run the full corpus under the de-circularized methodology. That's a paid run, and it's the next budgeted step — not something I'll fake with the old numbers in the meantime. When that re-run happens, I expect the Lane B numbers to be smaller than my v0.6-corrected ones, because the old rules were partly circular and the old matcher failed open. Both of those errors pushed the contract's score upward. Correcting them pushes it back down. So the v0.6-corrected figures — raw LLMs producing a contraindicated plan 32–51% of the time, dropping to 8–17% through the contract, a 3–5× reduction — still stand as published, but they carry an asterisk now: the Lane B side was measured with rules that had partly seen the answer key. The direction is real. The exact magnitude is pending an honest re-run. I'd rather publish a smaller number I can defend than a bigger one I can't. Safety governance you can't audit isn't safety. So I'm auditing mine, in public, including the parts that make me look worse — because that's the standard I want WPL held to, and the standard I think every serious AI health product will eventually be held to.

There's a second story from this stretch of work that isn't about the benchmark at all — it's about the live system. While the eval was getting more honest, production turned out to have a blind spot of its own: a whole class of clients whose exercises the safety layer didn't even know existed. That's the next post: My Safety Layer Had a Blind Spot in Production. The benchmark is open at github.com/gymbile/wpl-eval. Gymbile runs the contract in production.