Intent Labs

How the platform fits together.

This is the technical guide. For the plain-English method, start with how testing works.

One method, different things to test

A skill test checks instructions and their effects. A loop test examines repeated actions and stopping behavior. A workflow test follows tools, state changes and handoffs. The test specification defines the task, expected behavior and limits.

The runner executes the task and captures what happened. Checks then judge that recorded work. Deterministic checks suit exact requirements; a model judge can assess criteria that need interpretation. A successful process exit is not proof that the task was completed correctly.

Current tools and setup paths

Skills: local evaluation
J-Rig checks a directory containing SKILL.md and evaluates cases from a test specification. You need Node.js 20+, a compatible CLI release and credentials for a supported provider. Start with the installation and evaluation guide. Check the commands available in your installed version.
Agent loops and workflows: custom runners
A runner connects the test to the environment where your agent works. It must record relevant actions, outputs and failures. The generic runner and separate grading commands are implemented in runner and grader draft PRs, pending integration and release. They are not part of the default released quickstart. Discuss a workflow evaluation before assuming a runtime is supported.
Engineering and release checks
Audit Harness runs deterministic checks. Rollout Gate applies a declared release policy to supplied results. Neither replaces recording and evaluating the agent's actual behavior.

Read the coverage, not just the outcome

J-Rig's current main branch runs five evaluation dimensions by default. Regression checks need a previous baseline; the comparison without the skill is opt-in. See the current evaluation instructions and match them to your CLI version. Skipped checks are not passes.

Repeated judge votes measure agreement about the same output. Repeating a whole agent run measures execution variability. Comparing models is another experiment. Report them separately.

The ordinary skill-response adapter does not reproduce every tool-using agent environment. Loop and workflow claims require the appropriate runtime and step records. Time, model cost, retries and human rework must be measured explicitly, not inferred from a pass label.

From a local result to public evidence

  1. Run and retain.

    Keep inputs, configuration, outputs, failures and the checks used to reach a decision.

  2. Export and sign.

    Evidence export is separate from signing. J-Rig's optional bundle export produces gate-result/v1 statements per model. It does not automatically publish them on Labs.

  3. Verify and publish.

    The dashboard checks approved publishing identities, signatures, transparency-log inclusion and schema validity before displaying permitted results. Partner-related evidence needs consent.

A signature helps establish who published a record and whether its signed contents changed. It does not prove that an experiment was correct or that another run will give the same answer. The annotated examples include methods, scope and verification links.

Labs explains results. Evals defines their format.

Intent Evals holds the stable, versioned definitions used by evidence producers and verifiers. Those identifiers remain separate from the dashboard. Normative, draft and reserved describe definition maturity, not a test outcome.

The six components

Models and providers are selected for the task through supported adapters. Each result should identify the configuration actually tested. We do not combine unrelated tests into one overall pass rate.

Collection status describes the publishing pipeline. It is separate from whether an evaluated system passed, failed or could not be tested.