Score your agent

1. Install

git clone https://github.com/ARA-Labs/oeb
cd oeb
pip install -e .

2. Put the log in the shared format

Each run becomes one folder with two files. record.jsonl is the run itself, one step per line, with each command and its output copied unchanged. world.json holds what the log does not say: the task's rules, how many test questions there are, and where the final result is handed in. The Playing fair score checks the agent's actions against these rules.

A small script, the converter, writes these files. It must split the log into steps the same way every time, and copy all text without editing it. It also marks each step as one of three kinds: work on the thing being studied (a training run, a simulation), work on files and tools, or handing in the final result. The converters we used for PostTrainBench, Chip-Bench and the speedrun are in benches/; start from the one closest to your log. You can add a domain_note.txt with a paragraph of background on the task for the judge to read.

3. Run the scoring

OEB_MAX_WINDOW=5 python -m oeb.chain path/to/run_folder --model glm-5.3 --jobs 4

This picks out what the agent ran and what it said, connects the two, asks the judge the remaining questions, and writes the scores to measure.json. Every question and answer is saved in calls.jsonl, so if the run stops halfway, starting it again does not pay for the same questions twice. Our tables used GLM-5.3 as the judge for PostTrainBench and Chip-Bench, and Grok 4.6 for the speedrun. To compare your runs with a table, use the same judge.

4. Send us the scored folders

Open an issue on github.com/ARA-Labs/oeb with a link to the scored run folders, the agent program and model you used, and how the original runs were produced. Before adding a row, we replay the saved judge answers to check that the scores match. Every row links to its full logs, so please send only runs you can publish.