How to test agents with evals
Ship every agent with a versioned eval dataset and a baseline score that can only go up.
An agent that cannot be evaluated cannot be trusted. Every agent in the Catalog ships with a Git-versioned eval dataset and a baseline score between 0 and 1 — a commitment, not a hope: the baseline may only rise across versions.
Build the dataset
Start with golden cases: realistic inputs paired with the outcomes a correct run must produce. The Beta program expects a minimum dataset size per agent — on the order of ten golden cases — before an agent is promoted.
The eval gate
CI re-runs the agent's evals on every change to its descriptor — and on every change to a skill or bundle the agent declares, which re-triggers evals across all declaring agents. Regressions against the baseline are flagged in the pull request.
Note
Eval gating is advisory during the Beta and becomes blocking at GA — a regression will then block the merge.
Reading scores
Each agent's current baseline appears on its Catalog entry — shipped agents today carry baselines between 0.70 and 0.80. Treat a rising baseline as the agent's quality contract with its consumers.
Still stuck?
Tell us what you were trying to do and where it failed — screenshots and run IDs help. A human follows up.