- Question
- After GPT-6.1 Terra was released on 2026-10-26, which CUELO roles should it take compared with GPT-6.1 Sol and Opus 5.5?
- Setup
- Tested coding (Terra high vs Sol high on two TypeScript + Bun tasks, n = 1), image reading (Opus 5.5 low vs Terra low and high across 4 synthetic images, 3 runs), and Windows Character Map desktop control (Terra vs Opus, 2 runs each).
- Finding
- On large-sheet, Terra passed 199/200 tests in 128 s vs Sol's 198/200 in 1,348 s, and all 4 desktop runs matched. On mega-six Terra stopped after 4 of 6 packages (585/960 vs Sol's 957/960) and trailed Opus on small-text OCR (138/153 vs 149/153).
- Limits
- Each model ran each coding task once, so variance is unknown and Terra stopping early may be a tendency or chance. Image reading used one synthetic image set, and desktop control tested one easy task over two runs per model.
CUELO Evals
These pages collect measurements the CUELO maintainer ran while building CUELO. Each study answers one question under stated conditions.
How to read them
Tasks, models, tool versions and run counts differ between studies, so numbers do not compare across studies and there is no overall ranking. Each entry lists its question, setup, finding and limits. Raw run logs, accounts and machine details stay private.
Studies
-
-
Skim role candidate comparison
- Question
- This study compares DeepSeek V4.1 Flash 1017, Argon, Haiku 5.5, GPT-6 Luna, and GPT-6.1 Terra for CUELO's
skimrole, which reads multiple source files and answers questions about them. - Setup
- The main agent ran two static question sets covering five source files five times each, using the omp core 18.13.1
skimcall path with identical input bytes and rotated call orders. It judged responses against a pre-written rubric without blind review, accepting only raw JSON arrays as format-compliant. - Finding
- GPT-6.1 Terra, GPT-6 Luna, and Argon answered consistently with the sources across all judged items, whereas faster models DeepSeek V4.1 Flash 1017 and Haiku 5.5 produced some contradictory or unnecessary "unknown" responses. Terra's and Luna's latency ranges overlapped. The main agent recommended Terra or Luna and did not change the skim setting.
- Limits
- Two question sets repeated five times are not a general coding-model ranking, and the main agent did not verify provider build fingerprints. Network variance, cache state and different reasoning-setting mappings are mixed into the latency figures, so the gaps do not show cause.
-
Maker sub-agent code quality comparison
- Question
- Which of six Maker sub-agent implementer candidates writes code that is easier to change later?
- Setup
- Six models ran 3 times at high effort on a 200-test spreadsheet formula engine. Scoring followed the VulcanBench code-quality method: 108 code-quoting AI reviews (L1) by three judges from other vendors (MiMo V2.6 Flash, Muse Spark 1.3, Grok 4.6) and two real-change tickets (L3).
- Finding
- AI review split into a top three (GPT-6.1 Sol 76.2, DeepSeek V4.1 Flash 75.2, Opus 5.5 74.5) and bottom three (GPT-6 Astra 72.0, Daybreak Blue 70.3, Sonnet 5.5 68.4) with overlapping ranges inside each group. Real-change scores clustered at 98.8 to 99.6 (rank correlation ρ = −0.03 with L1).
- Limits
- An L3 ceiling effect leaves open whether AI review predicts how hard code is to change. The study used one task with three runs per model. Its ranking was the reverse of VulcanBench, where Sonnet came first, and counting only one judge changes the order.
-
Harness comparison: CUELO vs Claude Code vs Codex
- Question
- With the same model and the same coding tasks, does the tool around the model change the result?
- Setup
- 76 total runs across six coding tasks ranging from a one-feature fix to six independent packages, graded by hidden tests after the run. Values are medians of repeated runs, and costs are API-price conversions from 2026-09-25 because runs used subscriptions.
- Finding
- On the hard task with Opus 5.5, CUELO finished 42% faster than Claude Code with the same correctness. After fixing bottlenecks and re-measuring, CUELO's six-package delegation time fell from 25 min 39 s to 16 min 13 s, while on the one-feature fix its API-price cost was 43% higher than Claude Code.
- Limits
- The tables mark the runs labeled "solo" that used sub-agents. The page keeps before-fix numbers and two reverted attempts, and all cost figures are API-price conversions, not billed subscription amounts.