Ekamyth Agent OS docs
Added for agents / Done means done

hero eval

Grades how each model did on your agent's real work, and scores your own test suites against a baseline so a change that makes things worse is caught before you keep it.

What hero eval does

It answers two questions: is my choice of model working, and did my last change make my agent worse? The default view grades every model call that went through the model gateway in the last 30 days by what really happened: whether it completed, failed or was retried, and how long it took. It doesn't ask another model to judge. For your own tests, hero eval init writes a test suite (an EvalSet) that you edit and run. Each run runs every case several times and records a pass rate. hero eval gate compares the latest run of each suite with the baseline you saved, and exits 1 when a suite or a case got worse, so you can use it in CI.

Limit: a green gate means your suite passed the cases you wrote. What the cases cover is yours to decide. See Limits.

Synopsis for hero eval

hero eval [status] [--json | --details] [--only <value>] [--since <when>] [--all]
hero eval list [--json] [--only <value>] [--since <when>] [--all]
hero eval gate          (also: hero eval --gate, hero eval -g)
hero eval update-baseline   (also: hero eval --update-baseline)
hero eval init [<name>] [--json]
hero eval negctl <name>
hero eval trust <name>
hero eval coverage <job-id>
hero eval describe [--json]
hero eval help          (also: hero eval --help, hero help eval)

Run with no subcommand, it behaves as status. An unknown subcommand or flag is refused with exit code 2. hero eval demo needs an examples package that isn't installed, and is refused with demos are not included in this build.

Subcommands of hero eval

SubcommandWhat it doesReads or changes
statusHow each model did on real work in the last 30 days, then your suites' recent runs and the latest pass rate. This is the default.Reads
listOne line per recorded suite run, newest first, each reading Eval run · not measured · — · see hero eval. The suite and its score are on hero eval. With none, it prints (no objects in window).Reads
gateCompares each suite's latest run with that suite's baseline. Exits 1 if a suite's pass rate or any case's pass rate dropped. With no runs or no baseline it only advises, and exits 0.Reads
update-baselineSaves the most recent run as the baseline for its suite. Other suites' baselines are kept.Changes
init [<name>]Writes a suite file, $HERO_HOME/evals/<name>.py, to edit and run. If the file exists, it's left as it is.Changes
negctl <name>Advanced. Runs a negative control for a verifier you registered: feeds it a deliberately broken input and checks that it fails. Exits 0 when the verifier caught it.Changes
trust <name>Advanced. Says whether a verifier is trusted, that is, whether it has passed its negative control.Reads
coverage <job-id>Advanced. For a job with a done-contract, which criteria have an executed proof receipt.Reads
describeThe command's own description: subcommands, flags and output schema. Reads no data.Reads
helpThe command's help.Reads

Arguments for hero eval

Arguments for status and list

ArgumentTypeRequiredDefaultAllowed valuesWhat it does
--jsonFlagNoOffNonePrints {data, envelope} as JSON. See Example 4.
--detailsFlagNoOffNonePrints only your suites' status, with every shared number as n/d and the reasons a figure is partial. The model view is left out.
--only <value>TextNoOffviolations, blocked, failed, or a model nameShows only matching suite runs. Text view only. Suite runs record no model, so a model name matches none.
--since <when>TimeNoOffA UTC time, <n>h, <n>d or today, up to 30 days agoShows only suite runs since then. Text view only.
--allFlagNoOff: the newest 20NoneShows every suite run. Text view only.

--only, --since and --all apply to your suite runs, not to the model view.

Arguments for gate and update-baseline

ArgumentTypeRequiredDefaultAllowed valuesWhat it does
--gate, -gFlagNoOffNoneThe same as gate.
--update-baselineFlagNoOffNoneThe same as update-baseline.

Arguments for init

ArgumentTypeRequiredDefaultAllowed valuesWhat it does
<name>TextNomy-agentA file name without .pyThe suite's name and file name.
--jsonFlagNoOffNonePrints data.path, data.created and envelope.status (created or exists).

Arguments for negctl, trust and coverage

ArgumentTypeRequiredDefaultAllowed valuesWhat it does
<name>Verifier nameYesNoneA verifier in $HERO_HOME/negctl-verifiers.jsonThe verifier to test (negctl) or ask about (trust).
<job-id>Job idYesNoneA job with a done-contractThe job whose criteria to check (coverage).

How hero eval gets its answer

Model choice. The default view reads the record of model calls that went through the model gateway over a rolling 30 days, groups them by kind of work and model, and counts calls, failures and retries, with median time and tokens per call. It compares two models on the same kind of work only when each has at least 10 calls there. Where one fails noticeably more, it recommends keeping the work on the better one. Where two do equally well, it names the cheaper one, at public API list prices.

Your suites. The file hero eval init writes defines what done right means as a list of checks on your agent's output, a list of cases, and an agent() function you replace with your real agent. When you run the file with python3, it runs each case 5 times, prints a scorecard, and appends one result to $HERO_HOME/eval-results.jsonl: suite, passed, total, pass rate, flaky cases and per-case results.

The gate. gate reads the results and $HERO_HOME/evals/baseline.json. For each suite, it compares the latest run's pass rate with the baseline's, and each case's pass rate with the baseline case's. A suite fails the gate when either dropped, even if another case rose. A suite with no baseline yet isn't gated. A run counts as stale after 14 days.

Get started with hero eval

Before you start: Agent OS is installed and hero setup has run. These commands work the same on Mac and Linux.

  1. See how each model did on real work.

    hero eval

    You should see IS MODEL CHOICE WORKING and a table, or, with no calls yet, No model calls yet.

  2. Write a suite for your agent.

    This command will:

    • write $HERO_HOME/evals/checkout-agent.py, unless it exists.
    hero eval init checkout-agent

    You should see eval init — scaffold written to and the file's path, then three steps.

  3. Edit the file: replace agent() with your agent and set the checks. Then run it.

    python3 ~/.hero/evals/checkout-agent.py

    You should see a scorecard and results appended → the results file.

  4. Save this run as the baseline.

    This command will:

    • save the most recent run as its suite's baseline.
    hero eval update-baseline

    You should see eval baseline updated → and the suite's pass rate.

  5. After your next change, run the suite again, then gate it.

    hero eval gate

    You should see ✓ no regression for each suite, or ⚠ REGRESSION and exit code 1.

Example output of hero eval

Output from version 0.1.36rc13 on a new install, with no model calls and no suite runs.

Eval · model choice · rolling 30d (Oct 18–Nov 17 UTC)
  No model calls yet — each call your agent makes through the Ekamyth Agent OS
    is graded here by what really happened: completed, failed, retried, how
    long.
  Run: hero run <agent>

YOUR OWN EVAL SUITES
  eval: no runs yet · threshold 14d
  no runs recorded — run your EvalSet, then `hero eval`
  scheduled runs: hero schedule

Example output with calls and suite runs. The format is the command's; the names and numbers are examples. Trimmed where you see …

Eval · model choice · rolling 30d (Oct 18–Nov 17 UTC)
RECENT RUNS
  Nov 14 09:12 · checkout-agent · passed 10/10 · —
  Nov 12 16:40 · checkout-agent · ✗ failed 9/10 · 1 flaky · —

IS MODEL CHOICE WORKING
  ! 1,240 calls graded across 3 kinds of work · 4 failed · downgrade replays: none yet
  ! verified outcomes: none yet — graded on completion, retries and time; no
    done-contract checked these calls

HOW EACH MODEL DID ON REAL WORK
…
RECOMMENDATION
  → not enough calls to compare models yet on: …
  graded by real outcomes, not by another model · moves and savings: hero route ·
    your own EvalSet runs: hero eval --details

YOUR OWN EVAL SUITES
  eval: last run 2026-11-14 · pass 100% · threshold 14d
  scheduled runs: hero schedule

How to read hero eval output

The model view

PartWhat it shows
TitleEval · model choice · and the window read, such as rolling 30d with its dates.
RECENT RUNSYour suite runs, newest first, 20 at most: when, suite, passed or ✗ failed with passed/total, flaky cases, STALE (<n>d) when older than 14 days, and — because a suite run records no cost. Shown only when there are runs.
IS MODEL CHOICE WORKINGCalls graded, kinds of work, failures, and how many downgrade replays matched the requested model. ! marks any failure or mismatch; ✓ means none.
HOW EACH MODEL DID ON REAL WORKA table: work, model, calls · failed · retried, median time, tokens / call.
RECOMMENDATIONWhich model to keep a kind of work on, or a cheaper one that does it as well. Costs marked ~ are estimated at public API list prices.
YOUR OWN EVAL SUITESOne line: eval: last run <date> · pass <rate> · threshold 14d. With several suites, the rate is the lowest suite's latest run, named in brackets. A stale run reads STALE.

The gate

Example output. The format is the command's; the names and numbers are examples.

eval --gate — change-gate per suite (regression fails with exit 1)
  suite checkout-agent
    baseline · pass-rate 100%  ts=2026-11-14T09:12:00Z
    candidate· pass-rate 90%  ts=2026-11-16T10:05:00Z
    ⚠  REGRESSION in suite checkout-agent: pass-rate dropped 100% → 90% (Δ -10%)  — gate failed
       regressed case(s): case-2
  promote when intended:  hero eval --update-baseline

A suite that held reads ✓ no regression (Δ +0%). A suite with no baseline reads no baseline for this suite yet, not gated. With no baseline file, it prints eval gate — advisory · no baseline (run `hero eval update-baseline` after a run) and exits 0.

More hero eval examples

Example 1. To gate a change in CI

python3 ~/.hero/evals/checkout-agent.py && hero eval gate

The step fails on exit 1, when a suite or a case regressed. When you mean the change, save it with hero eval update-baseline.

Example 2. To see recent failed runs only

hero eval --only failed --since 7d

Example 3. To see your suites' status with its basis

hero eval --details

Output from version 0.1.36rc13 on a new install.

eval: no runs yet · threshold 14d
  no runs recorded — run your EvalSet, then `hero eval`
  scheduled runs: hero schedule
  partial_reasons: producer_absent:eval_run, rate_card_stale
  provenance: —
  all retained · as_of 2026-11-17T06:00:43.053043Z · snapshot snap_c6a0afa
    · query eval.runs

Example 4. To use the results in a script

hero eval --json

It prints data and envelope. data carries headline, runs, last_run, pass_rate, gate, next_step and model_choice (rows, calls, recommendations). envelope carries the window, as_of, freshness against the 14-day threshold, and partial_reasons.

Example 5. To check a verifier before you rely on it

hero eval negctl my-verifier
hero eval trust my-verifier

negctl prints caught ✓ (verifier failed on the plant — trusted) and exits 0, or MISSED ✗ (untrusted, cannot gate: …) and exits 1. trust prints <name>: trusted or untrusted with the reason. A verifier with no negative control reads untrusted — refused — untrusted: no negative control yet (run: hero eval negctl <verifier>).

Exit codes and errors for hero eval

CodeMeaning for this commandCauseWhat to do
0Done, or advice onlyA view printed; the gate found no regression or had no baseline; a baseline was saved; a verifier caught its plant or is trusted.Nothing.
1A truthful negative resultThe gate found a regression; negctl missed its plant; trust says untrusted; coverage found a criterion without proof.Read the output.
2RefusedAn unknown subcommand or flag, a missing argument, an unregistered verifier, no such job, or demo.See the errors below.
3A file can't be read or writtenThe results file or the baseline can't be read, or another process holds the baseline.Fix or move the file the message names, then try again.
ErrorCauseFix
eval <subcommand>: refused — <file> <reason>. Fix or move that file, then rerun your EvalSet.The results file can't be read, for example it isn't UTF-8. Exit 3.Fix or move the file.
eval --gate: refused — <file> <reason>. Fix it, or delete it and run `hero eval update-baseline`.The baseline exists but can't be read. Exit 3.Fix it, or delete it and save a new one.
eval update-baseline: refused — another writer holds <file> past <n> s; nothing was written. Rerun when it finishes.Another process is writing the baseline. Exit 3.Run it again.
eval update-baseline: no runs in eval-results.jsonl yet — run your EvalSet firstNo suite has run. Exit 0.Run your suite file.
error: verifier '<name>' not registered in <file>negctl for a name that isn't in the verifier list. Exit 2.Register it, or check the name.
error: plant input unreadable: <path>The verifier's broken-input file can't be read. Exit 2.Fix its plant_path.
error: no job '<job-id>'coverage for a job with no done-contract file. Exit 2.Check the id.
demos are not included in this buildhero eval demo. Exit 2.None.
eval has no flag '<flag>'An unknown flag. Exit 2.Check the synopsis.
eval has no verb '<word>'An unknown subcommand. Exit 2.Check the synopsis.

Files and environment for hero eval

NameKindPurpose
HERO_HOMEEnvironment variableRoot of Agent OS's files on your machine. Default ~/.hero.
$HERO_HOME/evals/<name>.pyFile, written by initYour suite. Edit it, then run it with python3.
$HERO_HOME/eval-results.jsonlFileOne line per suite run. HERO_EVAL_LOG names another file; set it the same way for your suite file and for hero eval.
$HERO_HOME/evals/baseline.jsonFile, written by update-baselineEach suite's baseline.
HERO_EVAL_STALE_DAYSEnvironment variableDays after which a run reads stale. Default 14.
$HERO_HOME/negctl-verifiers.jsonFile, read by negctlYour verifiers: for each name, the command to run (argv), the broken input (plant_path), its kind (plant_kind, default mutated-input) and an optional known-good input (good_path). HERO_NEGCTL_VERIFIERS names another file. No hero command writes it.

Limits of hero eval

LimitBehaviourCan you change it?What to do
Green means your cases passedThe gate checks the cases you wrote, nothing more.Yes, by adding casesAdd a case for each thing that must not break.
Which calls the model view seesOnly model calls through the model gateway, over the last 30 days.NoRun your agent with hero run.
Comparing modelsTwo models are compared on a kind of work only when each has at least 10 calls there.NoKeep working; the comparison appears as calls build up.
Verified outcomesCalls are graded on completion, retries and time. None is checked against a done-contract, so the view says verified outcomes: none yet.NoNone.
The gate without a baselineWith no runs or no baseline, the gate advises and exits 0, so it never fails CI before you've saved a baseline.YesRun hero eval update-baseline once.
Naming the regressed casesNeeds per-case results, which suites written by hero eval init record. Otherwise the gate says per-case results not recorded for this suite.YesUse a suite file from hero eval init.
Which run becomes the baselineupdate-baseline saves the most recent run of any suite, for that suite.NoRun the suite you mean to save just before you save it.
coverage needs a done-contractNo hero command attaches a done-contract to a job, so coverage has nothing to read for most jobs.NoNone.
Filters--only, --since and --all work in the text view only, on suite runs.NoUse --json and filter it yourself.

Checked against Agent OS 0.1.36rc14 (build 599c5e120c18) on 17 Nov 2026.