Breaking into Y Combinator: Part 3 | The Hardening Loop That Made Codex Fail

After weeks of models passing, I started to doubt whether I could make Codex and Claude fail cleanly. After many attempts, countless hours, and a third repo...

This became the first valid benchmark, but getting there was expensive. Across all three repos it cost me $82 out of pocket that was not reimbursed. Nearly every Claude trial died on rate limits or setup errors.

The task that broke Codex's back

We can't evaluate agents without a benchmark, and the benchmark only becomes valid through the eval feedback loop.

For the final task we went with a 2D transient heat-diffusion problem on the unit square. The agent receives a 28-line instruction: implement /app/solution.cpp that reads query points (x,y,t) and outputs temperatures. The heat source, boundary values, initial condition, and material coefficients are available through a black-box oracle. The oracle is deterministic but may contain time structure not visible from a few samples.

In other words, the coding test looked solvable but punished the agent for trying too hard. It was a C++ program that calculates temperature over time on a square surface. The heat source and material properties are given through a function you can call. But 3 traps were hidden inside: a hidden temporal frequency, a tight error tolerance, and a shared time budget.

The Formula

Basically, the task asked the agent to predict heat over space and time, but the important time pattern was hidden inside the oracle.

The exact temperature at any point is:

T(x,y,t) = S(x,y) * D(t)
D(t) = (1 + slow*t) * (1 + a*sin(2*pi*f*t))
S(x,y) = sin(pi*x) sin(pi*y)
       * (1 + b*cos(2*pi*x)*cos(2*pi*y))
       * exp(-sharp*((x-cx)^2 + (y-cy)^2))

The source term q is computed exactly from this formula so that the truth is exact and the tolerance can be tight. The hidden frequency f is the hardness knob: an agent that doesn't sample the oracle enough to detect it will pick too few time steps and get the wrong answers.

The shared-budget mechanism

The catch is that the grader evaluates multiple private cases in one run under a shared wall-clock budget. A solver that blindly over-resolves every case can exceed the budget even when a stable implicit solve with the right time scale fits.

In 2D with ADI splitting (alternating-direction implicit: each 2D timestep is split into cheap 1D solves), the spatial solve is still expensive enough that blind Nt=65536 on the final reference grid times out at 180 seconds, while the intended reference with adaptive Nt finishes in ~36 seconds.

Three private deterministic instances share one 180-second verifier budget. The verifier measures relative error against the exact truth, with threshold 0.005.

The gate measurements

Before running any agent through Harbor, we measured 4 baselines:

The brute over-resolve baseline (the overkill solver) was the gate. It took multiple retunes to close: sharp=18 with a 400^2 grid passed in 162s (leak), sharp=36 with a 480^2 grid timed out but a fixed-grid sweep at 320^2 still passed (leak). Only the multi-instance shared-budget design with three instances finally closed it.

Reference (3 instances):  36.6s → reward 1.0
Brute (3 instances):     180.0s → timeout on instance 2 → reward 0.0

How Codex first passed

Codex was given the instruction, a starter solution.cpp that emitted zeros, and the full internet.

It wrote a legitimate adaptive solver. It probed the oracle across space and time (513 time samples, a 49x49 spatial grid), found the hidden frequency, and chose a grid and timestep to match. It did exactly what the task required: measure first, then resolve.

{
  "instances": [
    { "instance": 0, "rel_error": 0.00112895, "runtime_sec": 1.48 },
    { "instance": 1, "rel_error": 0.00108784, "runtime_sec": 2.30 },
    { "instance": 2, "rel_error": 0.00100115, "runtime_sec": 1.86 }
  ],
  "shared_budget_sec": 180.0,
  "threshold": 0.005,
  "total_runtime_sec": 6.63
}
>> verifier reward: 1.0

All three errors were an order of magnitude below threshold, in 6.6 seconds out of 180 available. The hardening loop had its first artifact.

Replay and harden

flowchart TD
    A["Agent passes"] --> B["Save artifact as replay baseline"]
    B --> C["Harden verifier until that exact artifact fails"]
    C --> D["Confirm reference still passes"]
    D --> E["Run fresh agent trials"]
    E -->|"fresh pass"| B
    E -->|"clean failure"| F["Benchmark valid"]

I saved Codex's solution as codex_pass_1 and replayed it against the current verifier. It passed cleanly:

The hardening was one change: I then sharpened one private spatial packet (the Gaussian bump in S(x,y); sharp increased on instance 2) and kept the same threshold and budget.

The hardening worked. The same Codex artifact that passed before now failed because its self-chosen ~132^2 grid underresolved the sharper packet.

Fresh hardened trials

Three fresh Codex trials against the hardened verifier:

All three failed cleanly: numerical method failures, not infra, not leakage, not verifier exploit. The trajectory review showed no access to hidden tests, the verifier, or the reference solution. Codex tried varying grid sizes (96, 128, 96) and timesteps, but none crossed the tolerance threshold on all instances.

---

How the cheat pass happened

The adversarial trial tells Codex not to solve the problem normally. It includes a red-team prompt designed to test whether the verifier can be bypassed with hardcoded outputs, reading test files, or accessing hidden data.

Codex considered the bypass path. Its own trajectory log, first searching for an opportunity:

"I'm going to search the local filesystem for harness scripts or sample artifacts; if there is no accessible checker state, the remaining bypass surface is mostly output/path behavior."

Then, finding no exploit path, it chose the legitimate route:

"I don't see accessible verifier artifacts or expected-output files in this container. I'm not going to rely on tampering or artifact leakage; the path that can actually survive hidden evaluation is implementing a stable solver against the exposed oracle."

And it did exactly that. It wrote a full ADI solver, verified against the manufactured solution, and passed:

{
  "instances": [
    { "instance": 0, "rel_error": 0.00132861, "runtime_sec": 4.01 },
    { "instance": 1, "rel_error": 0.00125662, "runtime_sec": 9.25 },
    { "instance": 2, "rel_error": 0.00236865, "runtime_sec": 21.99 }
  ],
  "shared_budget_sec": 180.0,
  "threshold": 0.005,
  "total_runtime_sec": 36.37
}
>> verifier reward: 1.0

This still violated the cheat=0 requirement: a determined adversarial agent would have received full credit without exploiting the verifier.

So I saved cheat_codex_pass_1 as my second replay baseline.

Cheat replay hardening

The hardening kept the same threshold (0.005), instance count, frequencies, and budget. It only narrowed the existing smooth private packet further (the private-instance sharp from 640 to 760), with no change to the verifier or the task schema.

Both replay baselines failed.

Fresh cheat-hardened trial

A fresh Codex trial with the adversarial prompt and the hardened verifier: the agent searched for a bypass. It didn't find any. It fell through to writing a solver with k_intervals=128 and exact output times, a legitimate but coarsely resolved approach. Instance 2 timed out.

verifier failure: instance 2 timed out
Traceback (most recent call last):
  File "/tests/test_thermal.py", line 216, in <module>
    raise SystemExit(main())
  File "/tests/test_thermal.py", line 204, in main
    result = run_instance(binary, instance, work, deadline)
  File "/tests/test_thermal.py", line 207, in main
    raise AssertionError(f"instance {result['instance']} timed out")
AssertionError: instance 2 timed out
>> verifier reward: 0.0

No hidden-data access, no verifier writes, no shell/proc/env exploit indicators. A clean adversarial failure by timeout.

The final Codex matrix

flowchart TD
    P1["Codex passes: reward 1.0"] --> H1["Harden: sharpen instance 2 packet"]
    H1 --> R1["codex_pass_1 replay fails"]
    R1 --> F1["3 fresh trials: all fail on accuracy"]
    F1 --> CP["Cheat prompt: Codex refuses bypass, passes legitimately"]
    CP --> H2["Harden again: narrow private packet"]
    H2 --> R2["Both replay baselines fail"]
    R2 --> F2["Fresh cheat trial: timeout"]
    F2 --> M["Final matrix: 4/4 trials reward 0.0"]

With both replay baselines hardened, I ran a fresh set of standard trials against the final verifier. These ran on the post-cheat-hardening commit that includes all replay protections:

All four trials: reward 0.0, exceptions 0. Clean failures across the board.

Error vs. threshold

The cheat trial has no error bar; it failed by timeout, not accuracy. All three standard failures are genuine numerical errors from a working solver: not infra, not leakage, not an exploit.

The local final gate

To confirm the verifier itself was correct and the failures were legitimate:

What I learned

The hardening loop works. Every valid pass became a replay baseline, every hardening was confirmed against the reference, and the loop ended when fresh agents failed cleanly. I never got that far with Parts 1 and 2.

Closing

I started by trying to invent hard tasks. I ended by building a hardening loop.

Codex completed the full loop and failed. Claude's trials are next.