CUELO

CUELO Evals

These pages collect measurements the CUELO maintainer ran while building CUELO. Each study answers one question under stated conditions.

How to read them

Tasks, models, tool versions and run counts differ between studies, so numbers do not compare across studies and there is no overall ranking. Each entry lists its question, setup, finding and limits. Raw run logs, accounts and machine details stay private.

Studies

  1. GPT-6.1 Terra vs GPT-6.1 Sol and Opus 5.5

    Measured
    Language
    English, Korean
    Question
    After GPT-6.1 Terra was released on 2026-10-26, which CUELO roles should it take compared with GPT-6.1 Sol and Opus 5.5?
    Setup
    Tested coding (Terra high vs Sol high on two TypeScript + Bun tasks, n = 1), image reading (Opus 5.5 low vs Terra low and high across 4 synthetic images, 3 runs), and Windows Character Map desktop control (Terra vs Opus, 2 runs each).
    Finding
    On large-sheet, Terra passed 199/200 tests in 128 s vs Sol's 198/200 in 1,348 s, and all 4 desktop runs matched. On mega-six Terra stopped after 4 of 6 packages (585/960 vs Sol's 957/960) and trailed Opus on small-text OCR (138/153 vs 149/153).
    Limits
    Each model ran each coding task once, so variance is unknown and Terra stopping early may be a tendency or chance. Image reading used one synthetic image set, and desktop control tested one easy task over two runs per model.
  2. Skim role candidate comparison

    Measured
    Language
    Korean
    Question
    This study compares DeepSeek V4.1 Flash 1017, Argon, Haiku 5.5, GPT-6 Luna, and GPT-6.1 Terra for CUELO's skim role, which reads multiple source files and answers questions about them.
    Setup
    The main agent ran two static question sets covering five source files five times each, using the omp core 18.13.1 skim call path with identical input bytes and rotated call orders. It judged responses against a pre-written rubric without blind review, accepting only raw JSON arrays as format-compliant.
    Finding
    GPT-6.1 Terra, GPT-6 Luna, and Argon answered consistently with the sources across all judged items, whereas faster models DeepSeek V4.1 Flash 1017 and Haiku 5.5 produced some contradictory or unnecessary "unknown" responses. Terra's and Luna's latency ranges overlapped. The main agent recommended Terra or Luna and did not change the skim setting.
    Limits
    Two question sets repeated five times are not a general coding-model ranking, and the main agent did not verify provider build fingerprints. Network variance, cache state and different reasoning-setting mappings are mixed into the latency figures, so the gaps do not show cause.
  3. Maker sub-agent code quality comparison

    Published
    Language
    Korean
    Question
    Which of six Maker sub-agent implementer candidates writes code that is easier to change later?
    Setup
    Six models ran 3 times at high effort on a 200-test spreadsheet formula engine. Scoring followed the VulcanBench code-quality method: 108 code-quoting AI reviews (L1) by three judges from other vendors (MiMo V2.6 Flash, Muse Spark 1.3, Grok 4.6) and two real-change tickets (L3).
    Finding
    AI review split into a top three (GPT-6.1 Sol 76.2, DeepSeek V4.1 Flash 75.2, Opus 5.5 74.5) and bottom three (GPT-6 Astra 72.0, Daybreak Blue 70.3, Sonnet 5.5 68.4) with overlapping ranges inside each group. Real-change scores clustered at 98.8 to 99.6 (rank correlation ρ = −0.03 with L1).
    Limits
    An L3 ceiling effect leaves open whether AI review predicts how hard code is to change. The study used one task with three runs per model. Its ranking was the reverse of VulcanBench, where Sonnet came first, and counting only one judge changes the order.
  4. Harness comparison: CUELO vs Claude Code vs Codex

    Published
    Updated
    Language
    Korean
    Question
    With the same model and the same coding tasks, does the tool around the model change the result?
    Setup
    76 total runs across six coding tasks ranging from a one-feature fix to six independent packages, graded by hidden tests after the run. Values are medians of repeated runs, and costs are API-price conversions from 2026-09-25 because runs used subscriptions.
    Finding
    On the hard task with Opus 5.5, CUELO finished 42% faster than Claude Code with the same correctness. After fixing bottlenecks and re-measuring, CUELO's six-package delegation time fell from 25 min 39 s to 16 min 13 s, while on the one-feature fix its API-price cost was 43% higher than Claude Code.
    Limits
    The tables mark the runs labeled "solo" that used sub-agents. The page keeps before-fix numbers and two reverted attempts, and all cost figures are API-price conversions, not billed subscription amounts.