← Eval sets

Dogfood — the platform evaluates its own shipped skills draft

The Intent Eval Platform's job is to grade other people's skills. The fair test of that claim is whether it grades ours — on the same bench, by the same harness, held to the same signature discipline. This scorecard is that test: the 7-layer skill-binary-eval methodology run against Intent Solutions' own published CoreWeave skills. Both rows are now signed and anchored in the public Rekor transparency log — verdicts a stranger can verify without trusting us. The second row is a BLOCK of our own skill: it spent its first days held because its verdict flip-flopped across identical re-runs, and it earned its signature only after the judging was made noise-robust — 7/7 identical verdicts, with the judge's individual votes carried inside the signed evidence. The page says exactly how.

The board

Each row is one of our own shipped skills, measured by the platform. The attestation column is the load-bearing one: signed means an Evidence Bundle exists, is sigstore-signed, and is anchored in the Rekor transparency log at a citable index; held means the measurement exists but we are deliberately not signing it yet, for a stated reason. There is deliberately no composite score column — composing incommensurable findings into one number is the failure mode this platform exists to refuse.

Coverage, as recorded in each signed bundle's coverage field: these rows were measured on the trigger, functional, and behavioral layers of the 7-layer methodology; the regression and baseline layers were skipped, and the signed evidence says so explicitly rather than implying full coverage.

Provenance — signed, and verifiable by you

The first row — node-forensics (SHIP)

The coreweave-gpu-node-forensics verdict is not a claim you have to trust. Its Evidence Bundle was keyless-signed (Fulcio, no long-lived private key) by a reproducible GitHub Actions workflow identity — not a person's machine — and anchored in the public production Rekor transparency log at log index 2085904207. This is the same public-good infrastructure (Fulcio + Rekor) behind npm provenance and SLSA.

You do not have to trust us. Verify it yourself with cosign (no account, no keys) against the committed bundle:

cosign verify-blob \
  --new-bundle-format \
  --bundle coreweave-gpu-node-forensics.bundle.sigstore.json \
  --certificate-identity 'https://github.com/jeremylongshore/intent-eval-lab/.github/workflows/sign-dogfood-bundle.yml@refs/heads/main' \
  --certificate-oidc-issuer 'https://token.actions.githubusercontent.com' \
  coreweave-gpu-node-forensics.bundle.json

The second row — a signed BLOCK, with the votes inside

The coreweave-gpu-cost-leak-hunter verdict is also signed — and it is a BLOCK of our own skill. Its Evidence Bundle is the first on this board to carry the fold inputs, not just the folded decision: for every criterion, the number of judge samples, the agreement fraction, and the individual per-sample votes, plus the aggregation rule in force (5-sample majority voting, 0.8 blocker quorum). The signed claim is "the majority of these N recorded votes" — never "the judge said so" — and a verifier can re-derive the verdict from the signed bytes.

cosign verify-blob \
  --new-bundle-format \
  --bundle coreweave-gpu-cost-leak-hunter.bundle.sigstore.json \
  --certificate-identity 'https://github.com/jeremylongshore/intent-eval-lab/.github/workflows/sign-dogfood-bundle.yml@refs/heads/feat/sign-cost-leak-stable-block' \
  --certificate-oidc-issuer 'https://token.actions.githubusercontent.com' \
  coreweave-gpu-cost-leak-hunter.bundle.json

The identity string ends in the evidence branch ref that was checked out at signing time — that ref is frozen into the Fulcio certificate itself, so this command verifies permanently, whether or not the branch still exists in the repository. It is a historical fact about the signing run, not a live pointer.

What the signature attests — and what it does not. It proves the Evidence Bundle (by digest) is authentic and unaltered, and that this workflow identity signed it at the logged time. It does not declare a predicate URI — the skill-binary-eval/v1 predicate stays reserved, never under labs.*. And it does not claim the verdict is "correct" beyond the pre-registered criteria: a signature attests authenticity, not the truth of the science. The science stands on the eval, not the signature.

How a coin-flip became a signable verdict

This row spent its first days held, not signed: the same skill, same harness, same judge model returned SHIP on one run and BLOCK on another. A Rekor entry is permanent and public; signing a coin-flip would attest one noisy sample as if it were ground truth. Diagnosis found two noise layers, neither of them the skill:

1. Judge noise. The LLM judge was called once per criterion — un-seeded, and nondeterministic even at temperature 0. With ~10 subjective criteria at ~85% per-criterion reliability and a gate that blocks on any single blocker "no", roughly 80% of runs BLOCK from judge noise alone, even for a good skill (0.85¹⁰ ≈ 0.20).
2. Execution sampling. The skill-under-test was executed at the API's default temperature (~1.0), so the output being judged was a fresh random draw every run.

Measured live under both noise layers together: 6 BLOCK / 1 SHIP across 7 identical re-runs.

The fix, now in the harness: each judge criterion is sampled N times and majority-voted, the measured agreement fraction replaces the judge's self-reported confidence, a blocker "no" below the agreement quorum can warn but never noise-BLOCK, and execution is pinned to greedy decoding. Re-run under the full fix: 7/7 identical BLOCK, every criterion's votes recorded. The verdict is real, not noise — under reproducible conditions the skill leads with credential requests instead of the CFO-legible dollar headline its blocker criterion demands. Publishing a signed failing grade of our own skill is the discipline this board exists for: the earlier favorable run was the lucky draw, and the fix is now the skill's to make.

What promotes a row to signed

A held row becomes a signed row — with its verdict and a citable Rekor index — when three things line up, as they now have for both rows above (cost-leak-hunter is the worked example: held first, promoted only after its verdict reproduced 7/7):

1. the eval runs under a reproducible GitHub Actions workflow identity, not a person's machine;
2. its Evidence Bundle is keyless-signed through Fulcio (the same public-good infrastructure behind npm provenance and SLSA);
3. the signature is anchored in the public Rekor transparency log at an index anyone can fetch and verify with cosign, no account and no keys required.

Source and references

• j-rig 7-layer eval-set — the specification these skills were measured against
• Evidence Bench — j-rig-bench scorecard — the sibling results board whose signing discipline this page follows
• jeremylongshore/j-rig-skill-binary-eval — the harness that ran these evaluations
• jeremylongshore/claude-code-plugins — the public packs the skills under test ship in