How generated levels move through three test layers
- Automated playtesting
- Reinforcement learning agent
- Level generator
- Reachability
- Test coverage
- Held-out level
- Human playtest
Deterministic validation
checks reachability and hard structural constraints.
Diverse test agents
explore routes, outcomes and exploits across seeds.
Human sessions
establish readability, perceived fairness and experience.
Start with properties that can fail unambiguously
Before training an agent, check graph connectivity, required keys, spawn safety, legal slopes, objective existence and hard resource limits. These validators are fast, deterministic and explainable. They should reject impossible maps before expensive simulation. Store the seed, generator version and exact failure so the defect becomes a regression test instead of disappearing when the team rerolls the level.
Use more than one player model
A shortest-path bot finds different problems from an exploratory, cautious or combat-focused policy. Scripted agents are easy to interpret but limited to anticipated behavior. Reinforcement-learning agents can discover unexpected strategies, yet their reward and training distribution shape what they notice. Unity ML-Agents explicitly supports automated testing and design evaluation, but agent competence must be demonstrated on held-out levels rather than assumed from training performance.
Define the generator and its structural gates with the AI level-design workflow, before using agent behavior as evaluation evidence.
Measure coverage, outcomes and failure traces
Record completion, time, deaths, resources, visited regions, action traces and points where the agent becomes stuck. Aggregate by seed and play style; one average hides impossible maps and extreme exploits. Replay representative failures visually to distinguish a level defect from an agent defect. Generalization research shows why training and test levels must remain separate: an agent can memorize a distribution without demonstrating robust play on new layouts.
Calibrate automation against human sessions
Humans reveal signposting, surprise, perceived fairness, pacing and strategies outside the agent's action model. Instrument both tests with compatible events, then ask where automated outcomes predict or contradict player experience. Do not train toward one bot until every map suits that bot. Keep a human review gate for meaning and enjoyment, and update the agent suite when players repeatedly expose an unmodeled behavior.
Compare agent traces with human play evidence
Select twenty unseen generated levels and freeze their seeds. Run structural checks, a shortest-path agent and one contrasting exploratory policy. Save failure traces and visually replay outliers. Put five representative levels into a small human session using the same completion and route events. Compare where machine outcomes agree, disagree or miss a player problem, then add one regression test for each repeatable gap.
Automated level-test coverage
Rejects impossible structures and hard-limit failures
Explores outcomes and paths across retained seeds
Evaluates readability, fairness, pacing and enjoyment
Generated-level playtesting checklist
- Run deterministic structural checks before simulation.
- Retain generator version and every failing seed.
- Use distinct scripted or learned play styles.
- Separate training levels from held-out evaluation levels.
- Calibrate automated findings against human playtests.
Questions about automated playtesting for generated levels
Can automated playtesting prove a level is fun?
No. It can expose measurable structural and behavioral failures, while fun and perceived fairness require human evidence.
Why use several agents?
Different policies reveal different routes, risks and exploits; one agent can make the generator overfit to its behavior.
Should failed seeds be deleted?
No. Keep them as reproducible fixtures unless they contain data that cannot be retained.
Apply this worlds & systems guidance
Create candidates with explicit rules through: Define constraints before producing level candidates. Extend the same test discipline to large spaces with: Apply retained-seed testing to generated worlds. The strongest suite combines deterministic checks, diverse agents and observed human play.
