A Verdict With an Expiry Date: Making AI Eval Results Ask to Be Re-Measured

Executive Summary

I run agent-corp, my personal autonomous AI agent fleet. A daemon on a dedicated server polls a GitHub Projects board and runs each task as headless Claude Code inside a Docker container. Every week, the daemon sends me a quality report.

Every change to the fleet, whether a prompt clause, default, model setting, or context-window policy, is decided by a small experiment. The result is recorded as a markdown file in docs/evals/, with a verdict such as KEEP, ADOPT, or revert.

The weakness was not that the experiments lacked documentation. Each eval already stated, in prose, when its verdict should be measured again. The weakness was that nothing read those dates. A verdict could expire while continuing to steer the fleet.

I fixed that with a small, report-only convention: each eval carries a machine-readable Re-measure-by: YYYY-MM-DD line, optionally including the trial count and exact rerun command. A scanner places every document into explicit buckets and adds the result to the weekly report. It does not block work or create tickets. It makes stale evidence visible.

The pattern applies beyond agent fleets. Model behavior can vary under repeated runs, and production inputs or operating conditions can change over time. For any team making operational decisions from AI evaluations, a verdict should carry its own shelf life.

The Problem Was Not Missing Documentation

Before this change, the eval documents already contained the necessary information. They described the measurement window, included a “Remeasure protocol” section, and stated the condition under which a KEEP branch should be revisited.

That design looked responsible on paper. In operation, it depended on memory.

The date lived in prose. No daemon, script, or weekly report inspected it. A human had to remember that an experiment was due, find the document, interpret the protocol, and decide whether to run it. The mechanism had no way to distinguish “checked and nothing is due” from “nobody looked.”

Two incidents made the failure concrete.

The auto-compact context-window eval was re-dated on 2026-10-10, about 8 weeks after the window the document itself had set. It happened only because someone remembered.

The budget-clause baseline was merged on 2026-08-31 and promised a re-measure 7 days later, on 2026-09-07. It never happened. The verdict continued steering the fleet for more than five weeks past its own shelf life.

That is a dangerous property for an autonomous system. A stale recommendation is not merely old documentation. It remains an input to future behavior.

Why an Eval Result Needs a Shelf Life

Agent runs are noisy. In one series I examined, the same test on the same repository scored differently on different days.

That sequence does not prove that the underlying change was good or bad. It demonstrates something more basic: a single recorded number is a signal, not a proof.

The same applies to conventional machine-learning systems. A model can be evaluated against a reference distribution and later encounter changed inputs or changed relationships between inputs and outcomes. Operational monitoring therefore compares current behavior or data with a baseline and defines conditions for intervention. For language-model systems, repeatability and reproducibility are also distinct concerns: identical conditions may not always produce identical outcomes, while different setups can produce different results.

An eval document therefore needs at least two kinds of information:

The second item is not administrative decoration. It defines how long the evidence is allowed to influence decisions without another check.

A date does not claim that the verdict becomes false at midnight. It marks the point at which confidence must be refreshed.

The Design: One Line Next to the Verdict

I added a convention near the top of each eval document:

Re-measure-by: 2026-09-07

The line can also carry n= for the number of trials and cmd: for the exact command that reruns the measurement. The command is the one already named by the document’s own protocol; the scanner does not invent a new measurement procedure.

The full grammar, as the scanner’s own docstring states it, looks like this:

Re-measure-by: 2026-09-07 n=50 cmd: python scripts/quiet-defaults-remeasure.py

Here is how the line reads in two real documents after the backfill. The top of the budget-clause baseline:

Wall-clock budget clause baseline
Re-measure-by: 2026-09-07 cmd: this doc's "Remeasure protocol" section (snapshot pair, 7 days after the 2026-08-31 merge)

And the auto-compact document, which expires on a condition rather than on a date:

Re-measure-by: never  # condition, not a clock: a non-zero compact_count_7d in the weekly report (re-dated 2026-10-10)

The second example shows why the line is allowed to carry a comment: never on its own would say nothing about what does end the verdict, and the reader of the document deserves that sentence right next to the value.

The design has three deliberate properties.

The metadata is local

The re-measure date sits next to the verdict and within the document that explains it. A reader does not need to find a separate registry to understand the evidence.

This also keeps the metadata reviewable with the experiment. When the verdict changes, the re-measure policy is visible in the same diff.

“Never” is explicit

“Never” does not mean the result is universally valid. It means that the verdict does not expire on a calendar clock; something else ends it, and the document has to name what, as the auto-compact example above does.

The key distinction is between “this does not expire on a date” and “nobody recorded a date.” Those states must not collapse into one.

Missing metadata remains visible

A document with no line goes into a separate “no line” bucket. It is not treated as never-expiring.

That choice matters during adoption. Backfilling is incomplete by definition. If missing metadata silently means “safe forever,” the reporting mechanism hides the exact work it needs to expose.

The Scanner’s Contract

The scanner reads docs/evals/*.md and sorts every document into four categories:

An unparseable line is reported, never silently skipped. A malformed date is operationally different from a missing line: the repository attempted to declare an expiry, but the declaration cannot be interpreted.

Every overdue item is shown together with its own re-measure command where one is recorded. That turns the report from a warning list into a runnable handoff. The human still decides whether to execute it, whether the protocol remains appropriate, and whether the resulting verdict should be KEEP, ADOPT, or revert.

The parser is a handful of regular expressions. This is the actual set from the scanner module:

_LINE_RE = re.compile(
    r"(?im)^re-measure-by:[ \t]+(.+?)[ \t]*$")
_DATE_RE = re.compile(r"^\d{4}-\d{2}-\d{2}$")
_N_RE = re.compile(r"(?:^|[ \t])n=(\d+)(?=[ \t]|$)")
_CMD_RE = re.compile(r"(?:^|[ \t])cmd:[ \t]+(.+)$", re.DOTALL)

The line is matched case-insensitively at the start of any line in the document, cmd: takes everything after it so a command can contain spaces, and n= is pulled out before the date token is read. A date with the right shape but an impossible calendar value, such as 2026-13-99, fails date.fromisoformat and lands in the unparseable bucket instead of being dropped.

And this is the classification step itself, with the file-reading code around it left out:

match = _LINE_RE.search(text)
if match is None:
    no_line.append(name)
    continue
token, due, n, cmd = _parse_line(match.group(1))
if token.lower() == "never":
    continue  # explicit non-expiring verdict: its choice, not a gap
if due is None:
    unparseable.append((name, match.group(1).strip()))
    continue
entry = EvalDue(path=name, raw_value=token, due=due, n=n, cmd=cmd,
                status="")
if due < today:
    entry.status = OVERDUE
    overdue.append(entry)
elif due <= today.fromordinal(today.toordinal() + within_days):
    entry.status = DUE_SOON
    due_soon.append(entry)
else:
    entry.status = LATER
    later += 1

Two details carry most of the contract. due < today is a strict comparison, which is what makes a date equal to today "not overdue yet". And the due-soon edge is <= against today plus seven days, which is what puts day 7 inside the window and day 8 outside it. Both boundaries have their own tests, because an off-by-one here would quietly move every verdict a day earlier or later.

The scanner is intentionally report-only. It does not block a task launch. It does not file an issue by itself. It does not make an automatic change to the fleet.

That tradeoff is important. A hard gate would make stale evidence impossible to ignore, but it would also make an administrative date capable of stopping unrelated work. In my system, a cheap weekly list read by a human is preferable to a gate that treats every stale date as an emergency.

The Weekly Report Is the Integration Point

agent-corp already sends a weekly quality report. I added one block to that existing output.

The architecture is deliberately small:

There is no new schedule and no new service.

This is the block the first run put into the report on 2026-10-17. I left out the 48 file names in the last line and kept the count:

evals due (docs/evals verdict shelf-life scan):
  OVERDUE: 2026-08-31-budget-clause-baseline.md - due 2026-09-07 - this doc's "Remeasure protocol" section (snapshot pair, 7 days after the 2026-08-31 merge)
  no re-measure line (48 - never-expire is a choice, make it explicit): ...

The overdue line repeats the command the document itself names, so the next step is right there in the report. The last line is deliberately a little pushy: it does not say "48 documents are fine", it says that 48 documents have not made a choice yet.

This integration point also avoids creating a second operational calendar. The report is already where I inspect fleet quality. Adding eval freshness there makes the evidence lifecycle part of the existing operating rhythm.

The same pattern can work in other systems. A model registry, experiment repository, or evaluation dashboard can expose an “evidence freshness” section alongside quality metrics. The implementation does not need to be sophisticated; it needs to make stale decisions visible at the place where operators already review system health.

Unknown Must Not Look Clean

The scanner itself can fail. For example, the directory might be missing.

In that case, the report says UNMEASURED. It does not render an empty list.

In the renderer that is three lines, and they run before anything else is printed:

if view.get("scanned") == 0:
    lines.append("  UNMEASURED - no docs/evals directory found")
    return lines

The order matters. If this check came after the overdue and due-soon loops, a missing directory would fall through to the "nothing due; nothing overdue" line, which is exactly the false all-clear the design exists to prevent.

This distinction is one of the most important details in the design:

A failed check must not look like a successful check with zero findings. This principle applies broadly to monitoring, policy checks, data validation, and security scans. If a broken observer reports “nothing found,” operators learn to trust absence that was never actually established.

Case Study: The First Backfill

I backfilled the line into the five eval documents that already stated a date in prose, using exactly the dates their own text gave.

The first scanner run was on 2026-10-17. It surfaced the budget-clause baseline, the one whose header is quoted above, as genuinely overdue, 40 days after its own date.

That was the result I wanted from the mechanism. It did not repair the missed measurement, reinterpret the old verdict, or pretend that the baseline was invalid. It exposed the stale decision at the weekly checkpoint.

The other backfilled documents demonstrated why explicit conditions matter:

Without the separate “no line” bucket, these four documents would have looked indistinguishable from an eval that had never declared its policy. The backfill therefore tested both the data and the meaning of the convention.

The same first run also showed how much is still undeclared. The scanner read 54 eval documents; 48 of them carry no line at all. That is not a failure of the mechanism, it is the backlog the mechanism was built to make visible: every week the report names the first eight of them and counts the rest, until each one gets a date or an explicit never.

The limitation is equally clear: this mechanism does not perform the measurement. It reports that the measurement is due. A human still has to assess whether the command remains safe, whether the repository state is appropriate, and whether the result should change the fleet.

Verification: Testing the Observer, Not Just the Parser

I treated the scanner as operational code, not a convenience script.

The tests cover:

I also ran a mutation check. Deliberately breaking the overdue branch made two test cases fail. With the branch restored, all tests in the touched suites passed.

The mutation check matters because ordinary happy-path tests can pass while a critical branch is disconnected. The scanner’s value is concentrated in its classification rules. If the overdue path silently stops working, the report becomes decorative while appearing healthy.

Safety Boundaries Around the Agent Fleet

The expiry mechanism does not execute agents, alter code, or merge changes. It sits beside the existing fleet workflow.

The fleet’s task pipeline is:

The evaluation freshness report does not bypass those controls. It only identifies when a prior experiment should be reconsidered.

That separation is intentional. The agent execution path has its own review and approval boundaries, while the eval scanner provides evidence hygiene. Combining the two too tightly would make a stale evaluation capable of blocking unrelated execution, even when the code-review pipeline remains healthy.

The scanner also produces an audit trail through the repository’s markdown documents and weekly reports. A future reader can see the declared date, the verdict, the protocol, and whether the item appeared as overdue.

Challenges and Tradeoffs

Noisy measurements

Re-measuring does not remove the variance shown in the chart below; it creates a prompt to examine it.

The operational implication is that the protocol should record enough context to make repetition meaningful. The optional trial count and exact command help, but they do not replace judgment about repository state, model configuration, or external conditions.

Calendar expiry versus condition expiry

A calendar date is simple to scan but can be the wrong abstraction, and a condition is not something this scanner can evaluate. Five documents already say never, and the scanner does not evaluate any of their conditions. For auto-compact that is acceptable, because the counter its condition names already appears in the weekly report. For the others, whether the condition has fired is still a person's call: the convention makes the condition visible to a reader, not to a machine.

False urgency

Not every overdue result deserves immediate intervention. A stale experiment may concern a low-impact default, while another may affect a central execution policy.

Because the system reports rather than blocks, prioritization remains human. The report surfaces age; the owner decides severity.

Parser and repository failure

The failure states described earlier make a broken scan visible, but they also create a maintenance obligation: the document convention must remain stable enough that the scanner can interpret it. Renaming the field, or moving it into a front-matter block, would silently push every document into the no-line bucket. The 48 count would jump, which is at least loud, but it would be the wrong alarm.

Organizational ownership

Even a solo system has an ownership boundary: the person who merges a fleet change is not necessarily the person who remembers its future measurement. Making the date machine-readable assigns the reminder to the system while leaving the decision with the human.

In a larger organization, this boundary becomes more important. The line should identify the protocol and date, while the surrounding process should clarify who is responsible for rerunning it and who can change the verdict.

Cost and operational overhead

The scanner itself is cheap because it is report-only and runs as part of an existing weekly report. The cost appears when measurements are rerun: agent execution, model inference, reviewer time, and any human review required by the protocol.

That cost should be visible rather than avoided by allowing stale evidence to persist. Teams can prioritize re-measurements by the consequences of the verdict, but they should not confuse “expensive to rerun” with “valid indefinitely.”

A Practical Rollout Plan

I would implement this in phases.

Start with the existing corpus

Search eval documents for prose such as “re-measure,” “remeasure,” and “after.” Do not infer dates that the document never stated. Backfill only dates that already exist in the source text.

This keeps the first migration auditable.

Define the line grammar

Choose one exact field name, date format, and explicit value for non-calendar expiry. A plain line rather than YAML front matter matched what the documents already were: none of the 54 starts with a front-matter block, so one line could be added in the same commit as the verdict without restructuring any file. The important property is deterministic parsing, not elaborate syntax.

Build the buckets before building alerts

Write the boundary tests before the report block exists. A scanner that cannot explain why a document was classified will be difficult to trust.

Add failure states

Decide what a failed scan prints before deciding what a clean one prints. This is a small implementation detail with a large operational effect.

Integrate without blocking

Do not make task execution depend on a clean expiry list until the organization has observed the workflow and understands the burden. With 48 undeclared documents on day one, a blocking version of this scanner would have stopped my fleet the moment it shipped.

Review the policy after use

Once the list becomes part of the weekly routine, inspect which documents repeatedly expire, which conditions are better than dates, and which protocols are too costly to rerun. The metadata should evolve with the evaluation practice.

Actionable Takeaways

FAQ

Why should AI evaluation results expire?

AI runs can be noisy, and system conditions can change after an evaluation. An expiry date does not declare the original result false; it marks when the evidence must be refreshed before continuing to rely on it.

Should an overdue evaluation block an AI agent from running?

Not necessarily. In my fleet, the scanner reports overdue evaluations but does not block task launches or file issues automatically. This preserves execution continuity while making stale decisions visible for human review.

What should “never” mean in an evaluation document?

“Never” should mean that the verdict does not expire on a calendar date because it is governed by a condition or completed measurement window. It should be explicit; a missing expiry line belongs in a separate no-line bucket.

What happens if the expiry scanner fails?

The report should say UNMEASURED, not show an empty list. An empty result means the scan completed and found nothing due, while UNMEASURED means the system could not establish the state.

How can teams handle noisy AI eval scores?

Record the exact rerun command and, where useful, the intended trial count alongside the expiry policy. Re-measurement creates another observation; it does not eliminate variance, so the protocol and resulting evidence still require technical judgment.

Related reading