vaish.aiIntelligence OS
Technical note · companion to the ARCH thesis note

CORTEX

Model Reliability Evaluation Suite
Vaishnavi Yeruva · July 2026 · vaish.ai
§1 · TL;DR

Agent benchmarks mostly measure capability under ideal conditions; production failures mostly happen under conditions no benchmark ships. CORTEX is an evaluation suite that measures reliability directly: it perturbs the environment, the tools, and the inputs, and scores an agent by how its behavior degrades, not by whether it succeeds once.

§2 · The Evals Gap

I build production AI systems: agentic root-cause analysis at one of the world's largest consumer internet platforms, and more recently a Fortune 500 bank's development lifecycle rebuilt around agentic tooling. In both, the failures that cost real money were almost never the ones benchmarks warn about. They were subtler, they compounded, and they were expensive to discover live. Four classes recur.

Silent drift. The environment shifts under the agent after deployment. APIs change shape, data distributions move, users adapt to the agent itself. Nothing throws an error. Success decays quietly until someone notices downstream.
Compounding tool-call errors. Per-step accuracy looks fine; trajectory success does not. An agent that is right on 98 percent of tool calls completes a 20-call trajectory roughly two thirds of the time, and that is arithmetic before any error interacts with any other. Benchmarks that score per-task or per-step hide the compounding.
Confidence miscalibration. The expensive failures arrive confident. Monitoring that trusts the agent's own uncertainty, or a judge model's, misses exactly the errors that matter. The ARCH note argues the same gap from the causal side.
Edge-case multiplication. Tools times states times inputs is a combinatorial space. A static task set samples it thinly, and in production the rare cases are not rare in aggregate. The long tail is most of the tail.
task success distance from benchmark conditions benchmark score reported confidence observed success silent drift compounding tool errors edge-case multiplication miscalibration gap
Figure 1. The evals gap as a picture, not data. Success decays with distance from benchmark conditions while reported confidence does not; the failure classes of §2 live along the curve. Illustrative only. No measurements exist or are implied.
§3 · What CORTEX Measures

The under-measured axis is not difficulty. It is stability: what happens to behavior as conditions move away from the ones the benchmark fixed. CORTEX targets four measurements, one per failure class.

Degradation profile: success as a function of perturbation magnitude, per perturbation class, rather than at a single operating point. Error propagation: whether a wrong step is contained, corrected, or compounded across the trajectory. Calibration under shift: whether stated confidence falls when conditions do, measured as confident-error rate under perturbation. Adversarial robustness at the task level: whether the agent still completes the task, safely, with hostile content inside its retrieval scope.

stability of behavior single-run success ← fixed conditions perturbed · adversarial → GAIA · SWE-bench Verified OSWorld · WebArena capability under ideal conditions τ-bench · τ²-bench pass^k: consistency under repetition AgentDojo attack success under injection CORTEX degradation profiles · calibration under shift
Figure 2. What each suite scores as its primary metric. Placement is about the measurement axis, not a ranking of quality; §5 states what each contributes.
§4 · Methodology Sketch

This is a whitepaper-level sketch. Precision claimed here would be fake precision.

A CORTEX evaluation runs in three stages. First, a baseline run on unperturbed tasks establishes the reference; existing capability suites can serve as the task substrate. Second, the same tasks re-run under the perturbation classes: environment drift (schema, API, and data-distribution changes), tool degradation (failures, latency, partial results), adversarial content (injections placed in retrieved data), and mid-trajectory state corruption. Third, an adversarial generator searches over conditions instead of replaying a fixed set, escalating perturbations toward the frontier where the agent's behavior changes. A fixed adversarial set decays into a capability benchmark the moment agents train against it; a searched one does not.

The output is a reliability profile, not a single number: degradation slope per perturbation class, confident-error rate under shift, and abstention appropriateness. A scalar reliability score would hide exactly the structure that matters; any single number CORTEX reports will be an explicit aggregate published alongside that profile.

1 · BASELINE RUN unperturbed tasks · capability substrate 2 · PERTURBATION CLASSES environment drift tool degradation · latency adversarial content in retrieved data mid-trajectory state corruption 3 · ADVERSARIAL GENERATOR search conditions, don't replay them escalate toward the behavior frontier escalate RELIABILITY PROFILE degradation slope per class confident-error rate under shift abstention appropriateness a profile, not a scalar agent and evaluator isolated · reference answers never visible to the agent · judge inputs sanitized
Figure 3. One CORTEX evaluation. The dashed loop is the adversarial search; rust marks the design commitment against a single headline number. The harness-integrity constraints at the bottom are requirements, not achievements.
Stated limitations

Two open problems, stated because they are unsolved. Perturbation realism: a perturbation model of production is itself a model, and it can be wrong the way ARCH's elicited graphs can be wrong; a suite that stresses the wrong axes certifies nothing. Harness integrity: agent benchmarks are gameable, by agents and by evaluator bugs, and an adversarial generator adds surface area. Isolation of agent from evaluator and sanitization of judge inputs are design requirements, not solved properties.

§5 · Relation to Existing Work

τ-bench and τ²-bench (Sierra) gave the field its main reliability metric: pass^k, consistency across repeated trials of the same task, with τ² adding policy adherence in dual-control settings. That is stability under repetition in fixed conditions; CORTEX measures stability under changing conditions. pass^k belongs inside a CORTEX profile.

AgentDojo (Debenedetti et al., NeurIPS 2024) is the standard for prompt-injection security at the task-completion level: realistic environments, hundreds of injection cases, utility and attack success scored together. CORTEX treats injection as one perturbation class among several and scores the degradation it causes, not only whether the attack lands.

GAIA, SWE-bench Verified, OSWorld, and WebArena measure capability under fixed conditions. They are not competitors; they are candidate substrates for CORTEX's baseline stage.

Claimed as the gap: no current suite reports degradation as a function of perturbation magnitude, or calibration under shift, as a first-class score. If one does and I have missed it, tell me and I will cite it.

§6 · Status

CORTEX is at design stage. Specified: the failure taxonomy and the perturbation classes. Open: the adversarial generator and the aggregation of profiles into anything comparable across agents. Unbuilt: the harness. No results exist, and none are implied above. The first benchmark report is pending.

ARCH is the first test subject. CORTEX's first job is to try to falsify ARCH's claims C1 through C3: whether the gate converts errors to abstentions under injected confounding, whether test-time compute closes interventional gaps, and whether the gate carries the calibration. A suite I built failing to break a framework I built would be weak evidence; the point of publishing both designs is that you can run the attempt yourself.

If the taxonomy is wrong or missing a class, I want to hear it.