Difficulty that grows with the player: measuring every room with simulated players

How hard a room feels depends on how good the player is by the time they get there. Room 12 is played by someone with eleven rooms of practice, so it can ask for more than room 1 and feel just as fair. The usual shape is a sawtooth: each area starts easy with its new idea, gets harder up to a test room, and the next area drops back down.

We asked Claude Code to measure part 5’s 12 rooms for new, average and skilled players and reshape them that way. It rebuilt the game as 15 rooms in three areas. Measured by us, the old game was flat at 99–100% for the player who has reached each room. The new one dips once in each area.

Real measurements from our own runs: every room of both versions played 200 times per skill level in the real game, with replays of the recorded attempts. The thick line is a player who starts in room 1 with ±0.10 s of timing error and improves steadily to ±0.05 s by the last room. Beat labels come from Claude's plan of the game. Switch between before and after, or drag the skill slider.

The waves are shallow in the first two areas. Area 1 bottoms out at 97% in its new pit run and area 2 at 91% in its double jump test. Area 3 goes lowest: 73% in Get it back, one room before its test (79%), then 100% in the last room.

Claude's 15 rooms. There is no way to skip ahead: our scripted run reaches area 3 after 21 seconds and finishes all 15 rooms in 41.5 seconds with no deaths. Spikes restart the room you're in. We only added the start screen and loaded Phaser from this site.

Before and After are pass rates for the player who has reached the room. Beats are from Claude’s plan; it relabelled four old rooms, for example Goal (a test) became Pillar (a twist) now that a new room ends area 2. Skilled players (±0.02 s) pass every room at least 98% of the time. New players drop to 39% in Get it back and 45% in the spring test.

What Claude changed

Area 1 used to teach the jump and the double jump. Claude split it in two, so the game now has three areas of 4, 4 and 7 rooms: jump, double jump, spring. Each area ends on a test room. Two of them are new: a run over three spike pits with only two-tile steps between them, and a room where you hop onto a two-tile pillar without running off it, then jump and double jump across a wide pit. A third new room, Home, comes after the spring test: one bounce on safe ground and a walk to the flag.

It changed six old rooms. Long jump’s ledge is a tile higher and its pit a tile wider. The old Goal room became Pillar, with a second pit three tiles wider. Bounce and double jump has a ledge three tiles higher. Get it back has one spring instead of two and a ledge two tiles higher, and now comes before Spring in the pit, which also lost a spring and got a farther, higher ledge. The spring test was rebuilt as a long double jump onto a one-tile spring pillar, a bounce across to a second one, and a bounce and double jump up to the door. We compared the files: only the room data changed (grids, names and areas), and every movement setting is the same.

How we measured, and why Claude’s numbers differ

We used our own harness, not Claude’s. It runs the game in headless Chrome at a fixed 60 frames a second and plays each room’s route 200 times per skill level, with a random timing error on every press and release. Unchanged rooms kept exactly the routes we used before. For the nine rooms that are new or changed (Pit run, Long jump, Pillar, Double jump test, Bounce and double jump, Get it back, Spring in the pit, Spring test, Home) we wrote new routes and searched for the most forgiving timing, the way a player settles on the easiest way through.

Claude’s report gives different numbers: 70% at the double jump test, where we measured 91%, and 89% at the spring test, where we measured 79%. It concludes that area 3’s test is too easy. Part of the difference is how the two harnesses model a new player. In Claude’s harness, ±0.10 s is an even spread that never goes past 0.10 s. In ours, 0.10 s is the typical error of a bell curve, so about a third of our presses miss by more than that. The routes differ too. Both are simulations, and the two harnesses agree on the old game: flat, with nothing below 93%.

The session took 1 hour 16 minutes and cost $4.52. The first 45 minutes went into Claude’s harness: making it fast enough, and fixing a bug it found itself, where the room reload after a death fired in the middle of the next trial. The rooms took the other half hour.

A simulated player only measures timing. It can’t tell whether a room is confusing, whether anyone notices that the spring gives the double jump back, or whether the game is fun. Claude’s report says the same and asks for a real person to play area 1 and the two test rooms. We haven’t done that yet.

Update: the measuring now comes with the skill

Since Claude spent most of this session building its own harness, we put ours into the skill. Claude now adds a small hook to the game and writes a route for each room, and the skill’s script does the rest. It runs the simulated players, prints the table above and writes a chart you can open in a browser. Every run measures the same way, with a bell curve of timing errors, so numbers from different sessions can be compared. The skill now models the player improving with each room played (quickly at first, then levelling out near average) rather than by how far through the game a room is, so numbers from the current skill differ slightly from the chart above.

We ran this same prompt again in a clean session with the new skill, on the game before the reshape. Claude used the bundled script instead of writing its own, had its first full table after about two and a half minutes, and finished in 27 minutes for $2.89 (here: 1 hour 16 minutes, $4.52). Its reshaped spring area landed inside every target band by its own table, and our re-run with a different random seed agreed within a few points.

You need Node and Playwright for the script. If they are missing, the script prints the install commands.

Get the skill. The level rules work for any engine. The measuring script runs browser games, such as Phaser or plain canvas. You can also install all seven skills as a Claude Code plugin or replay this part from its starting files.

Recorded with a clean Claude Code: no personal skills, plugins or MCP servers. The session had platformer-levels and our earlier platformer-feel skill installed.