SPADE ♠: Self-Play in Adaptive Synthetic Executable Environments

Abstract & Motivation

Continuous self-improvement needs an ever-expanding pool of self-generated, diverse, adaptive goals. Hand-built environment pools stay fixed, so the goals stop adapting once the learner catches up to them.

SPADE is a self-play RL framework in which one LLM writes its own training environments and then learns in them. As the Environment Designer, it writes complete, long-horizon environments as executable Python implementing reset()/step(); as the Reasoning Agent, it learns to act in them. Through hint-based regret, the gap between the Reasoning Agent's return with and without a privileged hint, the Environment Designer targets environments at the edge of the Reasoning Agent's ability while keeping them solvable.

Adaptive Synthetic Environments Scaling

4,976 generated environments, one dot each, placed by what they are about. Click a dot to read or run it, drag the slider to replay training, toggle runs to compare. Ringed dots are the 190 the Reasoning Agent almost never solved.

Held-Out Benchmark Performance

Synthetic Game Environments

Every environment here is a Python program the Environment Designer wrote during training, and you can play it yourself. Press Play it right here on a card below, or open any environment and hit Run: a Python runtime starts in your browser and puts you in the same reset()/step() loop the Reasoning Agent trained on.

Two environments from one run, early and late, to show how far the Environment Designer moves. Step 4 leaks the answer in its opening line, and the Reasoning Agent wins 31 of 32 tries in under 4 turns. Step 304 asks for the best projection direction in a 5-dimensional feature space, runs 1,012 lines instead of 173, and burns all 12 turns on every one of its 32 attempts.

Beat the Reasoning Agent

Two environments the trained 30B-A3B agent never solved: 0 for 32 attempts on each in the released logs. Play them yourself; answers must follow the format the observation asks for, the same rule the Reasoning Agent faced.

Take the Controls

RotatingSokobanEnv is a spatial puzzle the Environment Designer wrote at step 113: push the three boxes (B) onto the targets (T) within 25 turns, while the whole grid rotates 90° every three turns. Each move you make reaches the environment's own code as a \boxed{go ...} action, the same interface the Reasoning Agent played through during training.

The source behind the link below is exactly what runs here, repaired in one line (a stale flag had stopped the rotation from repeating). Random placement leaves most boards unwinnable inside the turn limit, so Start and Shuffle deal only the 51 of the first 300 that a solver proved beatable, easiest first. The Reasoning Agent trained with no such filter.

Synthetic Tool-Use Environments

Beyond games, the Environment Designer also writes tool-use environments. Each one simulates a backend, exposes its tools in an OpenAI function-calling schema, and runs a multi-step user workflow that advances only when the backend state checks out. The 30B tool-use run generated 4,023 of them over 105 steps; two are below, with the Reasoning Agent's episode verbatim.

SPADE

Verifiable-reward RL draws its signal from fixed, hand-built pools of environments that stop adapting once the learner masters them. SPADE makes environment design itself a learnable component of post-training: because any computable MDP can be written as a program, the space of environments the Environment Designer can reach is bounded only by its coding ability. The designer is trained online, together with the Reasoning Agent, so the environment distribution shifts as the Reasoning Agent improves.

Hint-based regret

The designer's reward for an environment is the gap between the Reasoning Agent's average return with the privileged hint and without it. High regret marks the learning frontier, solvable with the hint but not yet without; low regret with high returns marks mastery; low regret with low returns marks an intractable environment. Unlike a pure adversary, which is free to make environments unsolvable, or a cooperative designer, which can inflate agent reward without teaching anything, hint-based regret keeps every environment both solvable and at the frontier.

Grounded, memory-seeded generation

Each round the Environment Designer conditions on passages sampled from a pretraining corpus (10k math, 5k science documents) and on an environment memory of high-regret past environments. The corpus keeps the reachable distribution broadening; the memory targets difficulty where the Reasoning Agent is learning fastest. Removing either costs about 6 suite-average points; removing designer training altogether drops self-play 12.6 points below the untrained base.

Qualitative Analysis

Ablation Results

Without the corpus, generation collapses

With the grounding corpus, the Environment Designer keeps writing new environments; without it, it gets stuck on one maze. These two columns are the same training steps from each run. Over the full run that comes to 0.68 distinct environments per 100 with the corpus, and 0.04 without.

Here is one of those mazes in full. The no-corpus run wrote this same program 413 times out of 865, 296 of them in a row, and this copy cannot even be won: the exit is walled off from the start. Play it and see.

Fixed-environment baselines saturate; SPADE keeps improving

Scaling Results

Citation

@misc{liu2026spade,
  title  = {SPADE: Self-Play in Adaptive Synthetic Executable Environments},
  author = {Bo Liu and Simon Yu and Yiding Jiang and Ao Qu and Andrew Zhao and Zichen Liu and Junsu Kim and Zijian Zhou and Seungone Kim and Tongzheng Ren and Mickel Liu and Hanfei Yu and Zhaorun Chen and Weiyan Shi and Paul Pu Liang and Luke Zettlemoyer and Yejin Choi and Natasha Jaques},
  year   = {2026}
}

×

scroll or click to zoom · drag to pan · double-click to fit · Esc to close