hero eval
Grades how each model did on your agent's real work, and scores your own test suites against a baseline so a change that makes things worse is caught before you keep it.
What hero eval does
It answers two questions: is my choice of model working, and did my last change make my agent worse? The default view grades every model call that went through the model gateway in the last 30 days by what really happened: whether it completed, failed or was retried, and how long it took. It doesn't ask another model to judge. For your own tests, hero eval init writes a test suite (an EvalSet) that you edit and run. Each run runs every case several times and records a pass rate. hero eval gate compares the latest run of each suite with the baseline you saved, and exits 1 when a suite or a case got worse, so you can use it in CI.
- Use it when you're about to change a model, a prompt or a tool and want to catch a regression, want to know whether your suites still pass, or want to see which model does each kind of work well.
- Use something else when you want the model moves and savings (
hero route), whether a job finished and what proves it (hero verdict), or one session step by step (hero trace). - Alias: none.
- Reads or changes:
status,list,gate,trust,coverage,describeandhelpread.update-baselinewrites the baseline.initwrites a suite file.negctlruns a verifier you registered and writes a receipt.
Limit: a green gate means your suite passed the cases you wrote. What the cases cover is yours to decide. See Limits.
Synopsis for hero eval
hero eval [status] [--json | --details] [--only <value>] [--since <when>] [--all]
hero eval list [--json] [--only <value>] [--since <when>] [--all]
hero eval gate (also: hero eval --gate, hero eval -g)
hero eval update-baseline (also: hero eval --update-baseline)
hero eval init [<name>] [--json]
hero eval negctl <name>
hero eval trust <name>
hero eval coverage <job-id>
hero eval describe [--json]
hero eval help (also: hero eval --help, hero help eval)
Run with no subcommand, it behaves as status. An unknown subcommand or flag is refused with exit code 2. hero eval demo needs an examples package that isn't installed, and is refused with demos are not included in this build.
Subcommands of hero eval
| Subcommand | What it does | Reads or changes |
|---|---|---|
status | How each model did on real work in the last 30 days, then your suites' recent runs and the latest pass rate. This is the default. | Reads |
list | One line per recorded suite run, newest first, each reading Eval run · not measured · — · see hero eval. The suite and its score are on hero eval. With none, it prints (no objects in window). | Reads |
gate | Compares each suite's latest run with that suite's baseline. Exits 1 if a suite's pass rate or any case's pass rate dropped. With no runs or no baseline it only advises, and exits 0. | Reads |
update-baseline | Saves the most recent run as the baseline for its suite. Other suites' baselines are kept. | Changes |
init [<name>] | Writes a suite file, $HERO_HOME/evals/<name>.py, to edit and run. If the file exists, it's left as it is. | Changes |
negctl <name> | Advanced. Runs a negative control for a verifier you registered: feeds it a deliberately broken input and checks that it fails. Exits 0 when the verifier caught it. | Changes |
trust <name> | Advanced. Says whether a verifier is trusted, that is, whether it has passed its negative control. | Reads |
coverage <job-id> | Advanced. For a job with a done-contract, which criteria have an executed proof receipt. | Reads |
describe | The command's own description: subcommands, flags and output schema. Reads no data. | Reads |
help | The command's help. | Reads |
Arguments for hero eval
Arguments for status and list
| Argument | Type | Required | Default | Allowed values | What it does |
|---|---|---|---|---|---|
--json | Flag | No | Off | None | Prints {data, envelope} as JSON. See Example 4. |
--details | Flag | No | Off | None | Prints only your suites' status, with every shared number as n/d and the reasons a figure is partial. The model view is left out. |
--only <value> | Text | No | Off | violations, blocked, failed, or a model name | Shows only matching suite runs. Text view only. Suite runs record no model, so a model name matches none. |
--since <when> | Time | No | Off | A UTC time, <n>h, <n>d or today, up to 30 days ago | Shows only suite runs since then. Text view only. |
--all | Flag | No | Off: the newest 20 | None | Shows every suite run. Text view only. |
--only, --since and --all apply to your suite runs, not to the model view.
Arguments for gate and update-baseline
| Argument | Type | Required | Default | Allowed values | What it does |
|---|---|---|---|---|---|
--gate, -g | Flag | No | Off | None | The same as gate. |
--update-baseline | Flag | No | Off | None | The same as update-baseline. |
Arguments for init
| Argument | Type | Required | Default | Allowed values | What it does |
|---|---|---|---|---|---|
<name> | Text | No | my-agent | A file name without .py | The suite's name and file name. |
--json | Flag | No | Off | None | Prints data.path, data.created and envelope.status (created or exists). |
Arguments for negctl, trust and coverage
| Argument | Type | Required | Default | Allowed values | What it does |
|---|---|---|---|---|---|
<name> | Verifier name | Yes | None | A verifier in $HERO_HOME/negctl-verifiers.json | The verifier to test (negctl) or ask about (trust). |
<job-id> | Job id | Yes | None | A job with a done-contract | The job whose criteria to check (coverage). |
How hero eval gets its answer
Model choice. The default view reads the record of model calls that went through the model gateway over a rolling 30 days, groups them by kind of work and model, and counts calls, failures and retries, with median time and tokens per call. It compares two models on the same kind of work only when each has at least 10 calls there. Where one fails noticeably more, it recommends keeping the work on the better one. Where two do equally well, it names the cheaper one, at public API list prices.
Your suites. The file hero eval init writes defines what done right means as a list of checks on your agent's output, a list of cases, and an agent() function you replace with your real agent. When you run the file with python3, it runs each case 5 times, prints a scorecard, and appends one result to $HERO_HOME/eval-results.jsonl: suite, passed, total, pass rate, flaky cases and per-case results.
The gate. gate reads the results and $HERO_HOME/evals/baseline.json. For each suite, it compares the latest run's pass rate with the baseline's, and each case's pass rate with the baseline case's. A suite fails the gate when either dropped, even if another case rose. A suite with no baseline yet isn't gated. A run counts as stale after 14 days.
Get started with hero eval
Before you start: Agent OS is installed and hero setup has run. These commands work the same on Mac and Linux.
See how each model did on real work.
hero evalYou should see
IS MODEL CHOICE WORKINGand a table, or, with no calls yet,No model calls yet.Write a suite for your agent.
This command will:
- write
$HERO_HOME/evals/checkout-agent.py, unless it exists.
hero eval init checkout-agentYou should see
eval init — scaffold written toand the file's path, then three steps.- write
Edit the file: replace
agent()with your agent and set the checks. Then run it.python3 ~/.hero/evals/checkout-agent.pyYou should see a scorecard and
results appended →the results file.Save this run as the baseline.
This command will:
- save the most recent run as its suite's baseline.
hero eval update-baselineYou should see
eval baseline updated →and the suite's pass rate.After your next change, run the suite again, then gate it.
hero eval gateYou should see
✓ no regressionfor each suite, or⚠ REGRESSIONand exit code1.
Example output of hero eval
Output from version 0.1.36rc13 on a new install, with no model calls and no suite runs.
Eval · model choice · rolling 30d (Oct 18–Nov 17 UTC)
No model calls yet — each call your agent makes through the Ekamyth Agent OS
is graded here by what really happened: completed, failed, retried, how
long.
Run: hero run <agent>
YOUR OWN EVAL SUITES
eval: no runs yet · threshold 14d
no runs recorded — run your EvalSet, then `hero eval`
scheduled runs: hero schedule
Example output with calls and suite runs. The format is the command's; the names and numbers are examples. Trimmed where you see …
Eval · model choice · rolling 30d (Oct 18–Nov 17 UTC)
RECENT RUNS
Nov 14 09:12 · checkout-agent · passed 10/10 · —
Nov 12 16:40 · checkout-agent · ✗ failed 9/10 · 1 flaky · —
IS MODEL CHOICE WORKING
! 1,240 calls graded across 3 kinds of work · 4 failed · downgrade replays: none yet
! verified outcomes: none yet — graded on completion, retries and time; no
done-contract checked these calls
HOW EACH MODEL DID ON REAL WORK
…
RECOMMENDATION
→ not enough calls to compare models yet on: …
graded by real outcomes, not by another model · moves and savings: hero route ·
your own EvalSet runs: hero eval --details
YOUR OWN EVAL SUITES
eval: last run 2026-11-14 · pass 100% · threshold 14d
scheduled runs: hero schedule
How to read hero eval output
The model view
| Part | What it shows |
|---|---|
| Title | Eval · model choice · and the window read, such as rolling 30d with its dates. |
RECENT RUNS | Your suite runs, newest first, 20 at most: when, suite, passed or ✗ failed with passed/total, flaky cases, STALE (<n>d) when older than 14 days, and — because a suite run records no cost. Shown only when there are runs. |
IS MODEL CHOICE WORKING | Calls graded, kinds of work, failures, and how many downgrade replays matched the requested model. ! marks any failure or mismatch; ✓ means none. |
HOW EACH MODEL DID ON REAL WORK | A table: work, model, calls · failed · retried, median time, tokens / call. |
RECOMMENDATION | Which model to keep a kind of work on, or a cheaper one that does it as well. Costs marked ~ are estimated at public API list prices. |
YOUR OWN EVAL SUITES | One line: eval: last run <date> · pass <rate> · threshold 14d. With several suites, the rate is the lowest suite's latest run, named in brackets. A stale run reads STALE. |
The gate
Example output. The format is the command's; the names and numbers are examples.
eval --gate — change-gate per suite (regression fails with exit 1)
suite checkout-agent
baseline · pass-rate 100% ts=2026-11-14T09:12:00Z
candidate· pass-rate 90% ts=2026-11-16T10:05:00Z
⚠ REGRESSION in suite checkout-agent: pass-rate dropped 100% → 90% (Δ -10%) — gate failed
regressed case(s): case-2
promote when intended: hero eval --update-baseline
A suite that held reads ✓ no regression (Δ +0%). A suite with no baseline reads no baseline for this suite yet, not gated. With no baseline file, it prints eval gate — advisory · no baseline (run `hero eval update-baseline` after a run) and exits 0.
More hero eval examples
Example 1. To gate a change in CI
python3 ~/.hero/evals/checkout-agent.py && hero eval gate
The step fails on exit 1, when a suite or a case regressed. When you mean the change, save it with hero eval update-baseline.
Example 2. To see recent failed runs only
hero eval --only failed --since 7d
Example 3. To see your suites' status with its basis
hero eval --details
Output from version 0.1.36rc13 on a new install.
eval: no runs yet · threshold 14d
no runs recorded — run your EvalSet, then `hero eval`
scheduled runs: hero schedule
partial_reasons: producer_absent:eval_run, rate_card_stale
provenance: —
all retained · as_of 2026-11-17T06:00:43.053043Z · snapshot snap_c6a0afa
· query eval.runs
Example 4. To use the results in a script
hero eval --json
It prints data and envelope. data carries headline, runs, last_run, pass_rate, gate, next_step and model_choice (rows, calls, recommendations). envelope carries the window, as_of, freshness against the 14-day threshold, and partial_reasons.
Example 5. To check a verifier before you rely on it
hero eval negctl my-verifier
hero eval trust my-verifier
negctl prints caught ✓ (verifier failed on the plant — trusted) and exits 0, or MISSED ✗ (untrusted, cannot gate: …) and exits 1. trust prints <name>: trusted or untrusted with the reason. A verifier with no negative control reads untrusted — refused — untrusted: no negative control yet (run: hero eval negctl <verifier>).
Exit codes and errors for hero eval
| Code | Meaning for this command | Cause | What to do |
|---|---|---|---|
0 | Done, or advice only | A view printed; the gate found no regression or had no baseline; a baseline was saved; a verifier caught its plant or is trusted. | Nothing. |
1 | A truthful negative result | The gate found a regression; negctl missed its plant; trust says untrusted; coverage found a criterion without proof. | Read the output. |
2 | Refused | An unknown subcommand or flag, a missing argument, an unregistered verifier, no such job, or demo. | See the errors below. |
3 | A file can't be read or written | The results file or the baseline can't be read, or another process holds the baseline. | Fix or move the file the message names, then try again. |
| Error | Cause | Fix |
|---|---|---|
eval <subcommand>: refused — <file> <reason>. Fix or move that file, then rerun your EvalSet. | The results file can't be read, for example it isn't UTF-8. Exit 3. | Fix or move the file. |
eval --gate: refused — <file> <reason>. Fix it, or delete it and run `hero eval update-baseline`. | The baseline exists but can't be read. Exit 3. | Fix it, or delete it and save a new one. |
eval update-baseline: refused — another writer holds <file> past <n> s; nothing was written. Rerun when it finishes. | Another process is writing the baseline. Exit 3. | Run it again. |
eval update-baseline: no runs in eval-results.jsonl yet — run your EvalSet first | No suite has run. Exit 0. | Run your suite file. |
error: verifier '<name>' not registered in <file> | negctl for a name that isn't in the verifier list. Exit 2. | Register it, or check the name. |
error: plant input unreadable: <path> | The verifier's broken-input file can't be read. Exit 2. | Fix its plant_path. |
error: no job '<job-id>' | coverage for a job with no done-contract file. Exit 2. | Check the id. |
demos are not included in this build | hero eval demo. Exit 2. | None. |
eval has no flag '<flag>' | An unknown flag. Exit 2. | Check the synopsis. |
eval has no verb '<word>' | An unknown subcommand. Exit 2. | Check the synopsis. |
Files and environment for hero eval
| Name | Kind | Purpose |
|---|---|---|
HERO_HOME | Environment variable | Root of Agent OS's files on your machine. Default ~/.hero. |
$HERO_HOME/evals/<name>.py | File, written by init | Your suite. Edit it, then run it with python3. |
$HERO_HOME/eval-results.jsonl | File | One line per suite run. HERO_EVAL_LOG names another file; set it the same way for your suite file and for hero eval. |
$HERO_HOME/evals/baseline.json | File, written by update-baseline | Each suite's baseline. |
HERO_EVAL_STALE_DAYS | Environment variable | Days after which a run reads stale. Default 14. |
$HERO_HOME/negctl-verifiers.json | File, read by negctl | Your verifiers: for each name, the command to run (argv), the broken input (plant_path), its kind (plant_kind, default mutated-input) and an optional known-good input (good_path). HERO_NEGCTL_VERIFIERS names another file. No hero command writes it. |
Limits of hero eval
| Limit | Behaviour | Can you change it? | What to do |
|---|---|---|---|
| Green means your cases passed | The gate checks the cases you wrote, nothing more. | Yes, by adding cases | Add a case for each thing that must not break. |
| Which calls the model view sees | Only model calls through the model gateway, over the last 30 days. | No | Run your agent with hero run. |
| Comparing models | Two models are compared on a kind of work only when each has at least 10 calls there. | No | Keep working; the comparison appears as calls build up. |
| Verified outcomes | Calls are graded on completion, retries and time. None is checked against a done-contract, so the view says verified outcomes: none yet. | No | None. |
| The gate without a baseline | With no runs or no baseline, the gate advises and exits 0, so it never fails CI before you've saved a baseline. | Yes | Run hero eval update-baseline once. |
| Naming the regressed cases | Needs per-case results, which suites written by hero eval init record. Otherwise the gate says per-case results not recorded for this suite. | Yes | Use a suite file from hero eval init. |
| Which run becomes the baseline | update-baseline saves the most recent run of any suite, for that suite. | No | Run the suite you mean to save just before you save it. |
coverage needs a done-contract | No hero command attaches a done-contract to a job, so coverage has nothing to read for most jobs. | No | None. |
| Filters | --only, --since and --all work in the text view only, on suite runs. | No | Use --json and filter it yourself. |
Commands related to hero eval
Checked against Agent OS 0.1.36rc14 (build 599c5e120c18) on 17 Nov 2026.