Measuring puzzle difficulty with bots
Difficulty is not something you can feel reliably in a game you designed. Sunny Sort measures it by having six different players play every level and reading the disagreement between them: each bot is built to fail in a specific way, so which one wins tells you what kind of level it is.
The six players
LevelAnalyzer runs all of these on every level and crosses the results.
| Player | What it does | What it reveals |
|---|---|---|
| Beam solver | the best line it can find | is the level winnable, with how much slack, using how much of the board |
FirstFit |
first legal move | if it wins, there is no puzzle |
Hoard |
always beside the fattest pile | if it wins from a corner, a dominant strategy is alive |
Greedy |
best move looking one ahead | if it always wins, the game is shallow; if it never wins, it is unfair or demands planning |
Spread |
always as far away as possible | a control: shows that placement has consequences |
Random |
random | the difficulty floor |
The verdict is the worst problem found, not an average:
impossible → trivial → dominated → harsh → good
A level that is beautiful in four dimensions and trivially winnable in the fifth is trivial. Averaging would hide exactly the thing worth knowing.
The 0-100 score then penalises wins by dumb bots, board use concentrated in one corner, and slack that is too generous; it rewards cascades and winning lines that genuinely need the whole board.
A metric that lied
The first version measured "cells touched" and it saturated at nearly 100% everywhere.
The reason is structural: a bot only loses when the board fills, so every loss ends up having touched every playable cell. The metric was measuring the losing condition, not the strategy.
Concentration is now measured on the solver's winning line and in the bots' individual choices, never on their totals. It is the kind of error worth writing down, because a saturated metric does not look broken — it looks like a consistent result.
A dumb bot winning is conclusive. A good bot losing is not.
An earlier report classified nine levels as deep because the greedy bot did not beat them, and I read that as depth. It was a loose conclusion: "greedy loses" is the absence of shallowness, not the presence of decision.
So I measured the thing directly. If a level demands planning, some moment must exist where one legal move loses the level and another keeps it alive. Probing levels 4-20, sampling turns across the whole game rather than just the opening:
| Metric | Result |
|---|---|
| Strategic depth (smallest lookahead that wins) | 1 in every level |
| Decision width (legal moves that keep the win) | 100% in every level |
| Forced turns (≤ 1 winning move) | 0% |
| Recovery after a non-optimal move | 100% |
There was no wrong move. At any point, anything the player did still won. Those deep levels were beating that particular greedy bot because of a weakness in its heuristic — it did not value keeping the top of a pile high — not because they required planning. A one-move bot with a better heuristic beat all of them.
That reading changed what came next. Three directions that looked like the obvious next step — feed choice, board geometry, cascade setup — cannot produce decisions while there is no pressure. A decision only exists when different choices lead to different outcomes.
What the measurements are worth
The point of all of this is that no design change goes in without a number behind it. The combined effect of the changes the measurements justified:
| Metric | Before | After |
|---|---|---|
| Average score, levels 1-24 | 55.2 | 81.8 |
Levels rated good |
11 of 20 | 21 of 24 |
| Impossible levels | 3 | 0 |
| Trivial levels | 5, accidental | 3, the tutorials, on purpose |
| Tutorial moves | 8-15 | 5-7 |
Every one of those numbers comes from the same command, which anyone with the repository can run:
dotnet run --project src/StackSort.Tools -- analyze --from 1 --to 24
The measurement is not a report someone wrote once. It is a tool that re-runs, which is what makes it possible to notice when a change makes the curve worse.