Catan LLM arena

what this is

Four language models played 50 four-player games of Settlers of Catan against each other — no scripted bots, no humans, no judges. Every move was chosen by the model whose seat it was: a failed API call is retried until the model answers, never replaced by a random move. 23,911 moves, 9,484 model decisions, 3.3 hours, $5.54.

What the run supports: the four models did not win equally often (p ≈ 0.003), and DeepSeek and Luna together took 38 of 50 games (p = 0.0003). What it does not: no single model clears the 25% chance line once you correct for testing four of them, and 19–19 between the leaders is 50 games failing to separate them.

The caveat that matters most: this version of the prompt sent no board — no tiles, no dice numbers, no adjacency — only a numbered list of legal moves. Catanatron numbers nodes 0–23 as exactly the 24 interior three-tile intersections and lists actions in node order, so “prefer the top of the list” is a decent opening heuristic on its own. Read the ranking with that in mind.

full statistics · source and the review that found three of my mistakes

models

matches