Breaking into Y Combinator: Part 1 | Hybrid Retrieval Fusion
Intro
"Every benchmark is an eval and every eval needs benchmark tasks"
This is a 3-part series based on agent evals and terminal benchmark design for AI agents in terminal environments, based on a take-home interview I did for an AI company backed by Y Combinator.
We had 7 days to complete the task.
I've included the original requirements, the tasks that failed and passed, and what I learned from testing them against frontier coding agents. Notes from LLMs and their thought process are also included in this deep dive. The reader should be aware
The task is based on the open-source Terminal Bench 3, and the test had to run through the TB3 CLI harness harbor.
The TB3 harbor harness builds the Docker environment, launches the agent, collects artifacts, runs the verifier container, and writes the reward. It pretty much handles the entire agent trial lifecycle.
We were not required to submit a PR for TB3. However, we could not use an LLM as the judge. The task had to be very hard, or close to impossible for the models, while also being fair. Think of something that might take a PhD student days to do.
Before running the tests, one of the first things I did was look for a guide. I found very good material on agent evals, but I didn't find one detailed enough for the work that I was actually doing:
building the actual benchmark task, testing it against strong agents, watching how they solved it, and then making the task harder without making it unfair.
The task
For the first run, I wanted to see how hard it was for the agents. The goal was to see what the agents were good and bad at. After reading and summarizing many Arxiv papers, and going back and forth with GPT, Claude/Fable, and GLM 5.2, I chose Python as the language since the TB3 (Harbor) ecosystem is Python-native.
Based on prior ML projects, I was somewhat familiar with retrieval-augmented generation, retrieval systems, embedding-based ranking, and reciprocal rank fusion, so I went with hybrid-retrieval-fusion as the task.
The agent had to understand hybrid retrieval + fusion to fix it. And that was where I messed up because it understood it well.
Prerequisite: the assignment constraints
The assignment had to go through a full eval pipeline: below are the full requirements.
These are the settings I used for the models. I ended up changing the speed from standard to fast for GPT-5.5.
| Model | Setting | What counted |
|---|---|---|
| Codex GPT-5.5 | Extra-high reasoning | Clean semantic failure under `/run` |
| Claude Opus 4.8 | Extra max effort | Clean semantic failure under `/run` |
What did not count as a valid failure:
API failures were the most common issue since I only had the pro plan. I did end up buying usage credits more than once, especially for Claude, since I was on a deadline.
Task Design
The task design, verifier, and iteration process were reasonably okay, but the weak point was the model-failure evidence: strong agents got far and fast.
The job was to repair the retrieval pipeline until the hidden verifier agreed with a reference implementation.
The retrieval terms:
| Term | Meaning |
|---|---|
| BM25 | The keyword-search side |
| Dense | The embedding/vector-search side |
| RRF | Reciprocal Rank Fusion; merges ranked lists by position, not raw score |
| Oracle | The reference solution should pass |
| Nop | Doing nothing should fail |
My thought process at that time was basically: the more you throw at it, the more it will fail. That was not correct in hindsight. I kind of knew it was a full-court shot, and I was not expecting it to land anywhere near the basket.
The intended behavior was:
- retrieve candidate lists from both modalities
- preserve opaque document IDs
- fuse by rank, not raw score
- use one-based ranks
- fuse before truncating to top_k
- sort deterministically
- write the required JSON output
The instruction had to state the contract. The instruction is below.
Hiding the RRF rule would have made the task unfair, so the difficulty had to come from tracing the implementation, not guessing hidden requirements.
The strongest design idea was the bug-masking chain: each fix exposed the next failure.
Raw-score mixing hid the RRF rank-base bug. Fixing rank fusion exposed truncation-before-fusion. Fixing truncation exposed document identity collisions. Fixing identity exposed nondeterministic tie-breaking.
The goal was to create a real debugging path where partial fixes produced partial progress without accidentally solving the hidden cases.
The bitmask graph mattered because it gave a way to audit difficulty. Each bug had to be load-bearing. A 4/5 fix should still fail hidden queries. The fixed state should be the only path to full reward.
Verification strategy
The verifier used two evidence banks.
| Bank | Purpose |
|---|---|
| Bank 1 | Compare the submitted output against frozen hidden expected rankings: exact match, or it fails |
| Bank 2 | Recompute RRF from emitted sub-rankings to catch fabricated fused output |
Bank 1 made sure the final ranking matched the hidden reference. Bank 2 made sure the output was internally consistent. That mattered because a retrieval task can be cheated if the agent only learns or fabricates final fused rankings. Recomputing from sub-rankings closes that path.
This part of the task was directionally right. It treated the benchmark as a measurement protocol, not just a puzzle.
Iteration process
The first risk was that a strong agent could ignore the planted bugs and rewrite the fusion layer from the instruction.
That would collapse the task from "debug a regressed service" into "implement RRF from a spec."
So the fix was not to hide the spec. TB3 instructions still needed to be clear. The fix was to make the integration surface more realistic: configuration, pipeline orchestration, ranking entries, serializers, candidate helpers, ordering helpers, and fusion over precomputed rankings.
A clean rewrite should still require understanding the data flow. Rewriting should not be forbidden, but it should not be the easiest path.
A passing clean rewrite would mean the task was too small or too directly specified.The second risk was hand-holding in the source. Comments and docstrings repeated the answer key too close to the broken lines: exact RRF formula, union-before-truncation, opaque ID warnings, deterministic tie-break prose.
That led to the standard:
Model failure analysis
The strongest model evidence was also the reason the task failed as a benchmark.
We used high-effort frontier settings for Codex with extra-high reasoning. We would have used Claude 4.8 with extra max effort, but since Codex passed, there was no reason to try Claude. At this point, I started planning the next idea.
For Hybrid, the clearest recorded result is that the extra-high reasoning got all the way through the intended path. It did not fail at setup, file inspection, implementation, artifact generation, or hidden verifier format. It reached the reward.
The run notes show how quickly that happened.
In one Codex run, the model started with rg --files, then went straight into models.py, fusion.py, pipeline.py, serialization.py, candidate_utils.py, config, CLI, and rankers. Another run opened fusion.py first, then checked the same surrounding cluster.
The third Hybrid Retrieval Fusion run was the clearest trace. At around 02:44:30 (that is 2 minutes, not hours), Codex read config, ordering, BM25, dense, and ran the visible search command. That visible output already showed fused scores in the raw-score range, not the small RRF range. Around 02:45:51, it read pipeline.py, fusion.py, ordering.py, candidate_utils.py, and config.py with line numbers. Around 02:46:06, it patched fusion.py and ordering.py.
TB3 fit
HRF was way too easy because the codebase was really small (single repo, 5 files, all bugs in the fusion layer).
For a frontier agent, it was easy to read, diagnose, and patch all 5 bugs in under 2 minutes.
Depth measures (hidden fixtures, masking, anti-cheat) helped hygiene, but they could not compensate for the lack of genuine scale.
The graph shows the uncomfortable result: the verifier gates can pass, and the task can still be too easy for the required model bar.
Final lesson
A verifier answers: A benchmark task also has to answer:What we decided to carry forward
- two-bank verifier
- frozen hidden expected outputs
- RRF recomputation from emitted sub-rankings
- bitmask partial-fix audit
- rewrite-path tracking
- oracle/nop proof before agent trials
- documented failure taxonomy
- careful distinction between protocol-specific evidence and broad model claims
The ideas rejected:
Conclusion
HRF was an easy and small codebase: single-layer bugs, fully stated spec. The agents could read and rewrite it in minutes. No amount of hardening without restructuring the task itself would have changed that.
This Hybrid Retrieval approach failed, but I definitely got something out of it. I learned that for the next version of the task, I had to make it harder in the failure landscape, not just cleaner in the verifier.
It was my first attempt, so I did not want to spend too much time without getting feedback on how the agents respond. The repo is open sourced at GitHub.
Part 2 will focus on why I moved to a thermal physics task: challenging agents to solve real physics problems with numerical resolution tradeoffs, transient heat diffusion, grid-resolution choices, shared time budgets across hidden instances, and tight accuracy tolerances.