CaseVBenchby COXIT
Release
V5 / October 16, 2026
Drawing set
119 pages / 9 projects
Human annotations
1,430 objects / 5 categories
Evaluation
F1 / IoU 0.50

ARCHITECTURAL DRAWING BENCHMARK

AI accuracy on
architectural drawings

CaseVBench tests how accurately AI locates cabinets, countertops, floor plans, elevations, and callouts. In v5, GPT-6 Astra leads 20 models with 92% overall F1.

Research by COXIT · In partnership with LVL10

Architectural drawing crop with blue floor-plan boundaries, green elevation boxes, and pink callout annotations.
FIG. 01 Floor plan, details, and callout annotations.Click drawing to enlarge

How the benchmark works

FIG. 02 / REFERENCE & PREDICTIONClick drawing to enlarge
Ground truth on the left and Claude Fable 5.1 predictions on the right. Both show one floor plan, four elevations, and seven callouts. This example is labeled F1 1.00.
A single-page example. Human annotations at left; Claude Fable 5.1 predictions at right. The supplied comparison reports F1 1.00 for this page, not the model’s overall benchmark score.
01

One full page in

Each model receives the complete drawing page as one image, with the same prompt. It returns labeled bounding boxes for five object types.

02

Compare with human markup

The reference set contains 1,430 hand-labeled objects across nine publicly available projects. Pages range from one object to more than 60.

03

Match at IoU 0.50

A predicted box must overlap its reference box sufficiently: intersection area divided by union area must reach 0.50.

04

Count misses and extra boxes

Precision measures how many predicted boxes are correct. Recall measures how many reference objects were found. F1 combines both.

Scoring details and limitations
LOCALIZATIONIoU = intersection / union
DETECTION SCOREF1 = 2TP / (2TP + FP + FN)

TP: matched boxes
FP: extra boxes · FN: missed objects

How to interpret the results

What 92% means. A 92% F1 score does not mean exactly 92% of objects were found, or that a takeoff is 92% complete.

Scope is specific. These are direct, full-page runs. Structured cropping pipelines and COXIT’s custom production systems are outside this leaderboard.

Drawing sets differ. Page density, object size, resolution, and scan quality affect performance. Rounded aggregate scores alone do not establish statistical significance.

Available evidence. The linked articles provide results and annotated examples. The downloads below contain aggregate results only; they do not include source drawings, annotations, or prompts. Recent releases include the API model identifiers from the evaluation export.

Model results

F1 balances correct detections, missed objects, and extra predictions. Scores run from 0 to 100; higher is better.

† Mistral Large 4.0: 117 of 119 pages processed. F1 retains all 1,430 ground-truth objects; cost and time summarize the 117 processed pages. The export does not identify the two missing pages.

F1 scores, % / Higher is better

Select a column to sort · Tied scores share a rank

Swipe across for all categories. Model names stay visible.

CaseVBench v5, October 16, 2026: F1 scores in percent at IoU 0.50, 119 expected pages. Overall F1 pools all categories. Mistral processed 117 pages; see the coverage note.
Model
01GPT-6 AstraOpenAI928284999496
02Claude Opus 5.5Anthropic796051919084
03Gemini 4 ArgonGoogle766938917581
04Gemini 3.8 FlashGoogle705920978668
04Claude Fable 5.1Anthropic706134988764
06GPT-5.6 SolOpenAI655324988857
07Qwen3.8-MaxAlibaba626032989047
08GPT-6.1 SolOpenAI605748956551
09GPT-5.6 TerraOpenAI565123948738
10Gemini 3.5 FlashGoogle534516969033
11Muse Spark 1.3Meta525221989023
12Gemini 3.1 Pro PreviewGoogle513910897837
12Claude Sonnet 5.5Anthropic514834968030
14GPT-6 SolOpenAI492511968336
15Claude Opus 5Anthropic405119968915
16Grok 4.6xAI305791690
17Mistral Large 4.0†Mistral AI2714378642
18GPT-6 LunaOpenAI2313882386
19Claude Sonnet 5Anthropic2110073474
20Grok 4.7xAI203081440

Overall F1 is pooled across all objects. It is not an average of the five category scores.

0–100 F1

Source: COXIT’s October 16, 2026 results export. V5 adds Gemini 4 Argon (76%) and Mistral Large 4.0 (27%), computed from pooled TP/FP/FN counts at IoU 0.50 and rounded to whole percentages. The previous 18 model records are unchanged. Sample results are excluded.

ACROSS BENCHMARK EDITIONS

How the best scores have changed

Each line shows a lab’s highest F1 in each published edition.

Dates refer to benchmark editions, not model launches. Each point uses the models listed in that edition; the leading model can change by category. A flat line means the best published score is unchanged, not that the model was tested again. Scores are rounded; lines do not represent measurements between editions. † Mistral’s v5 result has 117 of 119 pages processed, with the full ground-truth denominator retained.

Cost & speed

September–October 2026 measurements: median cost in USD and median time in seconds per processed page, lowest cost first. Mistral: 117 processed pages; other models: 119.
Model
GPT-6 Luna$0.00429.5 s
Gemini 4 Argon$0.011*9.8 s
Claude Sonnet 5.5$0.0205.5 s
Muse Spark 1.3$0.02038.4 s
Gemini 3.5 Flash$0.02615.5 s
Gemini 3.1 Pro Preview$0.03319.2 s
Claude Opus 5.5$0.0408.9 s
Mistral Large 4.0†$0.041*206.8 s
Qwen3.8-Max$0.055154.9 s
GPT-5.6 Terra$0.05632.0 s
GPT-5.6 Sol$0.057*45.2 s
GPT-6.1 Sol$0.06125.8 s
GPT-6 Sol$0.06824.3 s
Gemini 3.8 Flash$0.071*100.1 s
Claude Sonnet 5$0.07260.8 s
Grok 4.7$0.078185.9 s
Claude Opus 5$0.11237.2 s
Grok 4.6$0.134331.9 s
Claude Fable 5.1$0.14822.3 s
GPT-6 Astra$0.18628.2 s

Sources & data

Results checked
Read the paperView on GitHub

The paper covers the September 2026 evaluation of 14 models. The leaderboard includes subsequent benchmark updates.

COXIT. CaseVBench: AI Architectural Drawing Benchmark. v5, October 16, 2026. F1 at IoU 0.50; 119 pages, 1,430 annotated objects, nine projects. https://casevbench.com/

Research by COXIT in partnership with LVL10. This site presents the published results; the original benchmark and methodology article provide the source context. Cite the release and metric alongside any score.

Published results

Versioned score tables. V5 includes API identifiers, per-model measurement months and processed-page counts, with partial coverage and promotional discounts noted. V1–v4 remain archived as originally published.

The October export matches the earlier benchmark scope and prior aggregate records. It does not provide dataset or evaluator version hashes, run IDs, prompts, or raw per-page cost and latency records, so full run reproducibility is not established.