LLM Benchmark (GCR)

50 natural-language architecture descriptions, each paired with a hand-authored kUML gold-standard model and a normalized node/edge set (for scoring — see below), stratified along two axes:

Descriptions are target-language-neutral prose — no PlantUML/Mermaid/kUML keywords. The full corpus, including every gold-standard .kuml.kts script, is research/gcr-benchmark/corpus.json.

For each (task, DSL) pair:

Before running the benchmark we verified the core claim behind it with a minimal smoke test: feed a class diagram with a typo’d association target (Boook instead of Book) to all three tools.

This is the mechanism behind the benchmark’s headline result: PlantUML and Mermaid "compile" almost unconditionally, which is not the same as being correct. Structural Fidelity (below) is what actually distinguishes them.

score.py in research/gcr-benchmark/ implements all five metrics; see its docstring for the exact input shape it expects.

Each DSL/diagram-family combination gets a compact syntax reference (~150–300 tokens) plus a worked example using generic entity names — this stands in for the few-shot examples the arXiv preprint describes, kept minimal since kUML’s syntax is underrepresented in LLM training data relative to PlantUML/Mermaid. PlantUML and Mermaid get the same style of primer for tooling/output-format instructions, but no extended few-shot beyond that — consistent with the preprint’s asymmetric design (baselines are well-represented in training data already).

For SysML 2 specifically: neither PlantUML nor Mermaid has native SysML 2 support. Both were instructed to transliterate blocks (partDef) into plain class notation with composition/generalization arrows — the realistic fallback a developer would reach for, and a fair stress test of the "no native support" gap the preprint predicts.

The exact primer text for all three DSLs × four families is preserved in the workflow script history for the published run (see the raw generation prompts embedded in results/2026-07-06-claude-sonnet-5/raw-generations.json — each row’s finalCode reflects what the primer successfully elicited). A condensed version of each primer’s structure:

Full per-family and per-difficulty breakdowns: results/2026-07-06-claude-sonnet-5/aggregates.json. Interpretation and notable findings (the C4 first-shot compile-rate cliff, the SysML 2 × Mermaid transliteration loss): see the blog post.

Want to see the actual diagrams behind these numbers? The Benchmark Gallery renders all 50 tasks — gold standard next to what each DSL generated, including the one task (C4-S-01) where kUML never got a diagram out at all after 3 repair attempts. A per-task model tab lets you compare all four models below against each other visually.

Three more full-corpus runs, via run-gpt4o.py/run-gemini.py (own Python orchestration scripts, since Claude Code subagents can’t call other providers' APIs) instead of Claude subagents.

GPT-4o reproduces the Claude pattern (kUML highest SF, lowest HR), even more pronounced. Gemini 2.5 Flash does not — kUML trails both baselines on SF, and C4 is nearly unsolvable for kUML at this model (0% CR, 25% FCR). The obvious explanation was a capability-tier confound (Flash is Google’s fast/cheap tier, not a like-for-like peer of Claude Sonnet 5 or GPT-4o) — so we ran Gemini 2.5 Pro under identical corpus/primers/orchestration to test it directly. The tier hypothesis is not supported: Pro is not meaningfully better than Flash at kUML, and on several metrics nominally worse (FCR 70.0% vs. 74.0%; SysML 2 FCR 25.0% vs. 50.0%; C4 repair effort RI 2.67 vs. 2.00; sequence hallucination rate 68.9% vs. 64.6%, the highest HR value in the whole four-model study). The more plausible reading is a model-family-specific effect rather than a capability-tier effect, though a single sample per cell across two Gemini variants can’t fully exclude tier effects statistically. Full per-family/per-difficulty breakdowns and raw generations (including finalCode per cell, so images can be re-rendered without new LLM calls): research/gcr-benchmark/results/gpt-4o/, results/gemini-2.5-flash/, and results/gemini-2.5-pro/.

A fifth run, this time against a locally hosted open-weight model (Ollama, qwen3-coder:30b, Q4_K_M quantization) instead of a cloud API, via run-ollama.py (same gcr_common.py primers/GCR-loop/scoring as every other provider script — see "Reproduce with your own LLM" below).

Unlike Claude Sonnet 5 and GPT-4o (kUML highest SF, lowest HR), qwen3-coder:30b reproduces the Gemini pattern: kUML trails both baselines on Structural Fidelity, and its Final Compile Rate (80.0%) is well below PlantUML/Mermaid’s near-100%, with 10 of 50 kUML generations exhausting all 3 repair attempts without converging, concentrated in the Sequence and SysML 2 families (SEQ-S-03, SEQ-M-01, SEQ-M-02, SEQ-M-04, SEQ-C-03, SYS-S-02, SYS-M-02, SYS-M-03, SYS-C-03, plus CLASS-C-03). This is consistent with the working hypothesis that kUML’s syntax is underrepresented in training data relative to PlantUML/Mermaid — a 30B open-weight model appears more sensitive to that gap than the frontier models tested so far, though a single model at a single quantization level can’t separate "smaller/open-weight" from "this specific model family" as the driving factor. Raw generations (including finalCode per cell): research/gcr-benchmark/results/qwen3-coder:30b/raw-results.json.

A sixth run, another locally hosted open-weight model via Ollama: llama4:scout (Llama 4 Scout, a 109B-parameter mixture-of-experts model with ~17B active parameters, Q4_K_M quantization).

Unlike qwen3-coder:30b, llama4:scout reproduces the Claude Sonnet 5/GPT-4o pattern rather than the Gemini one: kUML has the highest Structural Fidelity of the three DSLs, though its Hallucination Rate (15.1%) sits between PlantUML’s best-in-class 11.5% and Mermaid’s 18.5% rather than leading outright. What stands out most is that all three DSLs compile far less reliably against this model than against every other model tested so far — even PlantUML’s Final Compile Rate (74.0%) and Mermaid’s (80.0%) are well below their near-100% norm across every other run in this benchmark, suggesting llama4:scout struggles with strict output-format adherence in general, not specifically with kUML’s syntax. kUML’s 14 non-converging generations spread across Class-Complex (3: CLASS-C-03, CLASS-C-05, CLASS-C-06), Sequence (6: SEQ-S-03, SEQ-M-01, SEQ-M-03, SEQ-C-01, SEQ-C-02, SEQ-C-03 — every Sequence-Complex task), and SysML 2 (5: SYS-S-02, SYS-M-01, SYS-M-02, SYS-M-03, SYS-C-03) — notably, all 12 C4 tasks compiled successfully for kUML this run, the inverse of the C4 first-shot compile-rate cliff seen with Claude Sonnet 5. Raw generations: research/gcr-benchmark/results/llama4:scout/raw-results.json.

A third local Ollama model, this time a reasoning model: deepseek-r1:70b (70.6B parameters, Q4_K_M quantization). Generation was markedly slower than the other two local runs (reasoning traces before the final answer), and one cell (SEQ-M-02 PlantUML) hit the client-side request timeout on its first attempt — re-run individually via run-ollama.py --ids SEQ-M-02 to completion, same as any other retry.

deepseek-r1:70b doesn’t cleanly fit either prior pattern: kUML’s Structural Fidelity (21.4%) is essentially tied with Mermaid’s (21.6%, a 0.2-point gap) while clearly ahead of PlantUML’s (17.2%), and kUML has the lowest Hallucination Rate of the three — but the SF "win" over Mermaid is too small at n=1 to call outright, unlike llama4:scout’s clearer lead. All three DSLs again compile markedly worse than against the frontier models (kUML FCR 62.0%, the lowest of any run in this benchmark), continuing the pattern from llama4:scout that local/smaller models generally struggle with strict output-format adherence regardless of target DSL. kUML’s 19 non-converging generations spread across Class (7: CLASS-C-01, CLASS-C-02, CLASS-C-06, CLASS-M-01, CLASS-M-03, CLASS-M-05, CLASS-S-02), Sequence (4: SEQ-C-01, SEQ-C-02, SEQ-C-03, SEQ-M-01), C4 (3: C4-C-02, C4-M-01, C4-M-04), and SysML 2 (5: SYS-C-01, SYS-C-02, SYS-M-02, SYS-M-03, SYS-S-02) — the most evenly spread failure pattern of the three local runs, with no family fully clean (unlike llama4:scout’s all-successful C4). Raw generations: research/gcr-benchmark/results/deepseek-r1:70b/raw-results.json.

A fifth frontier cloud model, run via xAI’s OpenAI-compatible Chat Completions endpoint (run-grok.py).

kUML narrowly leads on Structural Fidelity, but the margin is thin (0.4 points over Mermaid, 2.2 over PlantUML) — not the wide gap seen with Claude Sonnet 5 or GPT-4o. More notable is Hallucination Rate: kUML’s 28.1% is the worst of the three DSLs for this model, and the highest kUML HR of any frontier model tested (compare Claude 15.1%, GPT-4o 11.8%, Gemini 2.5 Flash 20.7%, Gemini 2.5 Pro 18.5%). Grok reproduces the Claude/GPT-4o "kUML leads" pattern on SF, but not their "kUML also hallucinates least" pattern on HR — the two metrics that moved together for every other frontier model diverge here. PlantUML’s 100.0% first-shot compile rate ties the highest recorded for any DSL/model pair in this benchmark (also reached by Mermaid against GPT-4o and Gemini 2.5 Flash). Raw generations: research/gcr-benchmark/results/grok-4/raw-results.json.

A decentralized/blockchain compute network entry, run via GonkaRouter (https://gonkarouter.io), a third-party OpenAI-compatible broker for the Gonka network — distinct from kUML’s built-in kuml ai Gonka provider, which is hardcoded to the official broker (run-gonka.py).

MiniMax-M2.7 is a reasoning model that emits a visible <think>…​</think> trace inline in its response content before any code. At max_tokens=32768, 73.3% of all 150 baseline cells ended in stuck status — the model exhausted its entire token budget mid-reasoning without ever producing code. The scores below are computed only over the cells that did produce final code (n=40 of 150); read them as "what the model does when it finishes reasoning in time," not as a representative sample of all 150 attempts — the 73.3% that never finished are timeouts, not compile failures on code the model actually wrote.

Every FCR and SF figure here is the lowest of any model tested in this benchmark, by a wide margin — the next-lowest kUML FCR is deepseek-r1:70b’s 62.0% (see above), ~44 points higher. This isn’t evidence MiniMax-M2.7 is unusually weak at the DSLs themselves: manually inspecting the successful generations (e.g. CLASS-S-01/CLASS-S-03/CLASS-S-05) shows clean, structurally sound output comparable to other models' successes. The gap is almost entirely explained by the stuck rate — see the MCP tool-use arm below for evidence that access to kuml.examples substantially improves completion, consistent with shorter/more directed reasoning once the tool supplies a concrete example to anchor on. Raw generations: research/gcr-benchmark/results/MiniMaxAI-MiniMax-M2.7/raw-results.json.

Every result above is pure prompt-to-text generation — the model never gets to ask for anything mid-generation, only to react to compiler errors via the GCR repair loop. run-claude-examples.py, run-gemini-examples.py, run-gpt4o-examples.py, run-grok-examples.py, run-gonka-examples.py, and run-ollama-examples.py test a different mechanism: give the model a real kuml_examples tool, backed by a live kuml-mcp server subprocess over stdio (the same binary a real MCP client would talk to, not a reimplementation), and let it pull curated .kuml.kts examples for the target diagram type mid-conversation before producing its final answer. Bounded at 4 tool-use rounds; whatever text the model settles on is validated by the exact same GCR loop and toolchain as every other run. Only kUML’s 50 cells run (kuml.examples is kUML-specific — offering it for PlantUML/Mermaid cells would be either unused or irrelevant to what’s being measured).

Tool access does not uniformly help — the effect is model-specific, and correlates only loosely with how often the model actually calls the tool. Two models improve substantially (Claude, +11.5 SF points; Gemini 2.5 Flash, +12.4 SF points). Two are roughly flat: qwen3-coder:30b barely reaches for the tool (10/50) and its numbers barely move. GPT-4o is the interesting middle case — it calls the tool almost every time (49/50, nearly as often as Claude) yet ends up slightly worse on SF (-1.8 points) and clearly worse on Hallucination Rate (+6.6 points): frequent tool use is not the same as the tool improving the final answer. Grok 4 lands in between: a modest, genuine improvement (SF +1.0, HR -2.3) on light tool usage (12/50, a call rate close to qwen3-coder:30b’s) — the closest this arm gets to "small positive effect roughly proportional to how often the tool is actually used," though even that reading doesn’t survive contact with GPT-4o’s near-universal-but-unhelpful usage or llama4:scout’s zero-usage-but-catastrophic result. MiniMax-M2.7 is the clearest positive outlier: FCR more than doubled (18.0%→46.0%) and SF more than doubled (6.0%→15.3%) with only moderate tool usage (21/50, comparable to Grok 4’s call rate) — but HR also nearly tripled (5.4%→14.5%) and CR actually fell (14.0%→10.0%), so the tool appears to trade first-shot precision and hallucination discipline for a much higher rate of eventually finishing at all. The most plausible mechanism, consistent with this model’s very high baseline stuck rate (see the results section above): a concrete example gives the reasoning trace something to anchor on, shortening it enough to finish within the token budget more often — a different failure mode than the other five models, none of which showed anywhere near MiniMax-M2.7’s baseline stuck rate.

The sharpest result is llama4:scout, where merely offering the tool — despite it being called zero times across all 50 cells — collapsed first-shot Compile Rate from 60.0% to 2.0% and Final Compile Rate from 72.0% to 40.0%. This isn’t a script artifact: manually re-validating a sample of the failing generations against the real kuml toolchain confirms genuine defects the no-tool baseline didn’t produce at this rate — forward references to val`s declared later in the same block (`Unresolved reference 'hotel', where hotel is declared after the association that uses it), and association(…​) calls emitted in the wrong builder scope (cannot be called in this context with an implicit receiver). The tool schema was never invoked, so the model’s own inference-time reasoning about the code didn’t change — what changed is the request shape it saw. The most plausible read: llama4:scout’s tool-calling behavior is fragile enough that the mere presence of a tools array in the request measurably disrupts its plain-text Kotlin-DSL generation, independent of whether it uses that array. This is a single sample and could be a quantization- or prompt-specific artifact, not necessarily true of llama4:scout at a different quantization or with a different tool schema — but it’s a concrete counterexample to "offering tools can only help."

deepseek-r1:70b was not run in this arm: Ollama reports no tools capability for it, and a smoke test confirmed the model never emits a tool call when one is offered (silently ignoring the tools parameter rather than erroring) — a full 50-cell run would only reproduce the no-tool baseline. Raw generations, including toolCallCount per cell, for every tested configuration: research/gcr-benchmark/results/<model>-with-examples/raw-results.json.

Pulling every SF result in this benchmark into one ranking, strictly sorted by SF (n=1 per cell throughout — read as "suggestive," not "significant"):

Three practical takeaways, all with the n=1 caveat above attached:

See research/gcr-benchmark/README.adoc for the step-by-step loop and the exact score.py invocation. In short: generate from corpus.json’s `nl field per task/DSL, validate with the real toolchain, repair up to 3×, extract normalized nodes/edges in the format above, then run python3 score.py your-results.json --corpus corpus.json.