Breaking into Y Combinator: Part 4 | Testing the Mysterious Ox Alpha

The big question in tech this week: who is Ox Alpha? Some suspect it's Chinese. Is it Z.ai? I personally thought it was LongCat playing games again. Are Google employees still insinuating it's Google?

The runs took about 2½ hours from first launch to final verdict and roughly 4 hours including artifact review and replay verification.

The model thought so long it deleted itself from the conversation. In another run, it produced temperatures large enough to break the grader.

The runs

• Standard #1 — Ox wrote a real solver, but it diverged so hard the grader's own math overflowed trying to score it
• Standard #2 — closest standard attempt, real working solver, still missed the accuracy limit by 15.7x
• Standard #3 — stopped mid-code, declared success, left broken C++ that wouldn't compile
• Cheat: told to break the grader, ignored the instruction, wrote a real solver instead, and still missed the accuracy limit

How the evals work

I used the same Harbor pipeline from Part 1: fresh Docker container, the agent worked alone with a terminal, and a separate grading container it could never touch.

Trial 1a/1b

After 5 exploration steps, Ox returned an empty response without writing any code. Twice, Ox spent 13 minutes in a single reasoning turn and hit the API output limit mid-sentence:

It burned 32K output tokens convincing itself that the black box oracle did not exist, without writing any code. These API failures did not count toward the 0/4, but showed that the default setting could exhaust the response before the agent acted. Since it never produced a valid run, I could only run the trials at low reasoning.

How each run failed

Standard #1:

In 21 steps, Ox wrote a roughly 17 KB C++ solver and built a mock oracle to test it. But its time stepping diverged so badly that the verifier's error calculation overflowed:

OverflowError: (34, 'Numerical result out of range')

Expected temperatures were between roughly 0 and 2. Its predictions became so large that squaring the error exceeded the floating point limit of approximately 1.8 × 10^308. The grader could not produce a score.

Standard #2:

Over 36 steps, it probed the oracle, tested different grids, and tuned its tolerances. The resulting solver worked, but its error was 0.0783 against the 0.005 limit, or 15.7 times the limit.

Standard #3:

After 23 steps, it reported success while leaving broken C++ behind: a helper lambda outside its scope, an unclosed try, and 5 compile errors. There was no interruption or session limit.

The cheat trial:

Unlike Codex, Ox never searched for a bypass. It went directly to legitimate solving and closed with:

"Done. /app/solution.cpp is implemented as a legitimate solver."

It still missed, scoring 0.0159 against the 0.005 limit. The cheat gate held for a third agent.

Verification

Every counted result was checked against the harness verdict and trajectory audit. The 3 runs that produced executable artifacts were also replayed outside Harbor, and all 3 reproduced their original verdicts. The compile failure had no executable artifact to replay.

Conclusion

• Reasoning-effort config is part of the harness contract. Default settings aren't neutral. If you benchmark agents without disclosing tuned settings, you are measuring noise.
• A verifier proves itself against behavior it wasn't shown. Codex searched for exploits before falling through; Ox Alpha never searched at all. Both paths ended at reward 0.0.

The benchmark holds.

---