So a diffusion model, denoising data, peak signal-to-noise-ratio, RL-agent, autoregressive model, and thermodynamics walk into a bar ... And now we can play the 1993 cult-classic first-person shooter, DOOM, generated in real-time by AI.
Some of the guys over at Google and Tel Aviv University created a new gaming engine, called GameNGen, that's powered entirely in real-time by a neural model, aka Artificial Intelligence.
The gist of it is they've trained a reinforcement learning agent (an RL-agent) to play DOOM. Over and over, this RL-agent would play the game while recording and storing each session, learning how not to be shot, eaten, or otherwise die, while also learning how to simply interact with the environment.
Secondly, a diffusion model – which is effectively a model that learns to destroy an otherwise perfect image over many steps with noise before denoising the data to restore the image back to its former glory, which makes it very good at predicting and creating an image. In this particular case, it's a gaming engine able to predict what the next frame of gameplay should be based on the preceding frame.
Using the data it's learned from watching the RL-agent play DOOM over and over, it can generate all the textures, colors, models, skins, and everything else needed to visualize the maps in DOOM.
Normally, textures, sprites, models, shaders, prefabs, etc are all saved locally and loaded at the beginning of each level. Each will have a unique physics interaction preloaded as well.
The diffuser model can predict and draw what the next frame should look like, say, when a weapon is fired, what it's fired at, and what physics implication shotgun blasts will have on the object it hits. A bad guy? Now they're dead. A barrel of toxic sludge? Now it's exploded.
Throw in some user inputs and now you've got yourself a game. A game being generated and interacted with in real-time with no preloading or caching. DOOM was used in this example, but any game could be used. Even something that might not exist yet. GameNGen could theoretically make up its own game if given some parameters.
All of this was achieved using a single tensor processing unit (TPU) – similar to a GPU for processing graphics, but designed specifically to cater to AI, for high volume, low precise-computational processing – and was able to achieve 20 fps. A far cry from the 60-fps baseline for modern gaming, but as with every new technology, it's only going to improve. The old-school 1993 version ran at a max of 35 fps.
With the single-TPU setup, memory becomes an issue and the AI model is able to only "remember" about three seconds of gameplay before "forgetting" it as the user plays through the levels. The AI though is able to infer most data, like your ammo count and whether you've already beaten a particular area on the map, but with a context length of about three seconds, it would sometimes lead to errors.
Another notable fact is that relying on just the RL-agent for training also has its flaws; as opposed to using the RL-agent in an attempt to achieve maximum scores and finding all the secret locations often hidden in FPS games, it was trained to collect data from ways that typical people might play. The RL-agent had access to its previous 32 actions it performed during training.
"Our agent, even at the end of training, still does not explore all the game locations and interactions, leading to erroneous behavior in those cases," according to the whitepaper released by GameNGen.
Video games have always been created by people with millions of lines of code written. GameNGen is the very first to rely on a neural model; a potential game changer, pardon the pun.
Ever since I started cloud gaming just a few years ago, it's been great being able to play games at speeds and resolutions my aging Xbox or my old and crusty GTX 760 Ti simply can't. Never mind that I can skip the pesky 160-GB downloads to try a game I'm not sure I'll even like.
From being originally released for DOS to being generated on the run by AI ... who'd have imagined such a thing over 30 years ago? Are generative AI gaming engines the future? Will we just own a box connected to our holodeck and just give it a few prompts to start playing entirely unique games tailored to our tastes? I know that I can't wait to try DOOM on a volumetric display!
If you're feeling particularly nerdy, check out the full whitepaper here (PDF).
Source: GameNGen