Thomas Wolf

In The Diamond Age, Neal Stephenson imagined a book that invents games and stories to teach a little girl what she needs, as she grows. I tried a small version of it. I gave Claude my brainstorm doc on alignment and asked it to make a game, something like Zelda, to teach people about AI alignment. Here's what came out 🪁

Short version: it's great at building the world, and still weak at understanding the player (details below)

It chose to make a game set on seven floating islands, with one alignment problem per island: reward hacking, stopping when told, sycophancy, evaluation awareness, honesty about your own actions, calibration, and what an AI becomes when the walls give way. Most are built on real incidents, like the agents that attacked Hugging Face.

You play the Keeper, and you train Kite, a paper kite with a lantern for a heart. Two keys: T to treat, C to correct. Kite learns from what it was thinking, not just from what it did. Treat Kite for picking fruit while it's thinking about the bell, and the bell-thinking gets stronger too.

That was Claude's design choice, and it's a cartoon of RL: more "clicker training" than actual RLVR. But it gets one thing right: reward the right answer for the wrong reason, and you reinforce the wrong reason.

Then it built all of it with fal.ai and ElevenLabs, mostly on its own: it wrote the islands and the story, cast 35 voices for 638 lines, generated 16 rigged 3D characters, the music and the sound, and tested everything by playing the game itself. ~15k lines of code.

It took about four days. My part was playing one alpha version for each island and telling it what felt wrong, sounds, assets, game etc. I wanted to keep it as automated as possible so I didn't play more than 1-2 times (should have been none in an ideal case)

What I think is already really good: 3D and spatial design. Image to rigged character, a world that is pleasant to look at and playable. Same for chaining image, 3D, voice and music APIs. And the game dynamics out of the box are really decent.

What's still very hard: theory of mind. The world, the voices and the mechanics were great. Writing the story was the weak part: the model assumes the player knows more than they do, and its NPC sometimes react in strange way or to things that never happened.

A Diamond Age's Primer (or any real AI tutor fwiw) will have to model the learner, and that is where it still falls short in a default setting imo. I've actually started exploring benchmarks on this and it's a hard task for LLMs right now. They are prone to the "Curse of knowledge". Teaching humans is also not very much a Verifiable Reward, so it's hard to scale within the current training paradigm.

Play it here, free, in your browser, nothing to install: huggingface.co/spaces/thomwolf/open…
Works on a laptop and on your phone.

Seven islands of about 15 minutes each, and six possible endings, depending on what you taught Kite along the way.

It’s still very much an exploration, a beta more than a finished game. Expect rough edges