Games

The Mandalorian's "Unreal" sets: A new age of game-based filmmaking

The Mandalorian's "Unreal" set...
Disney's The Mandalorian has been mostly filmed in a real-life video game environment using fascinating new filmmaking techniques
Disney's The Mandalorian has been mostly filmed in a real-life video game environment using fascinating new filmmaking techniques
View 6 Images
Real cameras shooting in an Unreal environment
1/6
Real cameras shooting in an Unreal environment
The incredibly versatile set of The Mandalorian – giant LED screens form the walls and ceiling
2/6
The incredibly versatile set of The Mandalorian – giant LED screens form the walls and ceiling
Half the Mandalorian's ship was built on set, so the actors could interact with it, while the back half was generated digitally
3/6
Half the Mandalorian's ship was built on set, so the actors could interact with it, while the back half was generated digitally
The incredible digital set is 75 feet in diameter and 21 feet tall
4/6
The incredible digital set is 75 feet in diameter and 21 feet tall
Disney's The Mandalorian has been mostly filmed in a real-life video game environment using fascinating new filmmaking techniques
5/6
Disney's The Mandalorian has been mostly filmed in a real-life video game environment using fascinating new filmmaking techniques
While shooting, the LED backgrounds generate 3D views that respond to the position of the camera
6/6
While shooting, the LED backgrounds generate 3D views that respond to the position of the camera
View gallery - 6 images

Filmmaking is entering a new era, powered by the technology that's made video games some of the most compelling entertainment in recent history. It's not green-screen CGI as we know it, either – it's the creation of whole digitally projected sets.

Disney's The Mandalorian is the latest Star Wars spinoff to find massive cultural success, and one of the key challenges for this production has been figuring out how to achieve Star Wars-level visual effects and locations on a streaming series budget. Luckily, timing lined up beautifully with the development of some freaky new production techniques that have revolutionized the filming process.

Epic locations and backgrounds are crucial to a show like this, but location shooting is expensive, tricky and highly dependent on the weather and time of day. So Lucasfilm's Industrial Light & Magic VFX division decided to go with CG sets. Green screen shooting, though, can be a pain. Lighting and camera movements need to be tracked, then used to generate the final composited shots – another expensive and time-consuming process that also hinders the ability of actors, directors and film crew members to have a good idea of exactly what they're creating in real time.

Enter North Carolina's Epic Games, creators of the ubiquitous Fortnite game – better known to the older among us as the guys who made the equally impactful Unreal back in 1998. But the games themselves are only a part of the story; the Unreal Engine that powers them has become the most successful engine in history, and its ability to create 3D scenery, objects, lighting, atmospheric effects and architecture has seen it become a go-to graphics engine behind not only a slew of games from various developers, but for Virtual Reality creations and now filmmaking as well.

In a system that first debuted in 2019, Epic has created a warehouse-scale production studio that allows filmmakers to shoot live inside environments created in the Unreal Engine. It works by projecting the 3D digital environment onto giant LED screen walls and ceilings in a studio, and it can switch between generic views that allow an actor to feel part of a scene, and views that change the digital projection to match the perspective, focal length and focus position of a camera.

Real cameras shooting in an Unreal environment
Real cameras shooting in an Unreal environment

That means you can move the camera around as much as you like during shooting, be it handheld or on a jib or dolly, and the backgrounds will behave like you're shooting on location. But, of course, it's better than location, because you can set the time of day, weather conditions and lighting exactly how you like them and not have to touch anything up in post. You're capturing it all in camera.

What's more, you can see exactly what you're capturing in camera, no compositing required, and if you don't like where a mountain or a tree or a moon or a house is sitting, you can use the power of the Unreal Engine to drag and drop them anywhere else you want. Everything, including lighting, shadows and atmospheric effects like fog, will change immediately to accommodate, and that means reflections in shiny objects on set will be dynamic and perfect as well.

The lighting alone is a big deal. The digital world literally lights the physical space. If you move the sun in the background, or even decide to switch between day and sunset and night, you don't need to compensate for that by sending gaffers running about to change lighting gear. The lighting on the actors will match the lighting of the backgrounds, because the LED backgrounds are the lighting. And if you want to tweak the lighting at all, you can create virtual light sources of any color virtually and off-camera.

You can see the process in this excellent video from the Unreal team below.

Behind the Scenes with UE4’s Next-Gen Virtual Production Tools | Project Spotlight | Unreal Engine

For The Mandalorian, ILM built a cylindrical CG projection set like this, with a 75-foot (22.8-m) diameter and a 21-foot-high (6.5-m) ceiling, and the benefits immediately became obvious. Switching locations from Iceland to Scotland to entirely imaginary worlds no longer required a bunch of plane tickets and a massive logistical effort. Actors found themselves able to move, and gesture and point as if they were really standing on location. Directors found themselves able to work on dawn scenes all day, dictating exactly what they wanted the scene to look like rather than being at the mercy of the natural world.

Then things started to get creative. The team was so enamored of the incredible flexibility and speed of the system that they began shooting interior scenes in it as well as outside environments. Moving vehicle shots became a breeze. They began building physical set elements inside the virtual world, taking advantage of the dynamic lighting and backgrounds. They'd build half the Mandalorian's ship on set, so the actors could interact with it, and have the back half generated digitally. Watching the final product, you'd never know.

While shooting, the LED backgrounds generate 3D views that respond to the position of the camera
While shooting, the LED backgrounds generate 3D views that respond to the position of the camera

The upshot is a faster, leaner production – but also one that can get super creative with locations, lighting and environments; just what you want for a Star Wars piece. The Mandalorian looks and feels amazing, and it seems clear this technology will make an enormous impact on the coming generation of TV and streaming video content.

What about movies? Not so much. Many, if not most, blockbuster movies are screened in 3D these days. This Unreal Engine 4 technology can only generate a digital background for one camera at a time, so binocular shooting can't be supported. Hollywood will still need its green screens and giant battalions of CG effects artists if it wants to work in 3D.

So, we might be headed for a situation where 2D video and streaming content starts looking vastly more interesting and creative than the movies, while doing it a ton cheaper and faster. A lot of the best writing and acting talent is already moving across to small-screen productions, and this kind of technology could prove another telling blow to the world of cinema.

Enjoy a video from The Mandalorian's production team below.

The Virtual Production of The Mandalorian, Season One

Source: Industrial Light & Magic

View gallery - 6 images
6 comments
Stephen Zadarnowski
Wow great tech. Everything covered! Great story, thanks.
Miro
From day to day, beautiful people are less and less. Life becomes more and more expensive, because abilities of people are less and less. To find a good looking and acting somebody for a movie nowadays costs a lot. So it is cheaper to produce virtual heroes, no doubt the guys from Hollywood can count.
Bob809
Absolutely incredible to see how they do this, stunning technology and implementation. Just hope the 'wrong kind' of people get a hold of this... damn, they already have it.
DavidB
Good article! Loz rocks.
nick101
I've seen this coming for years, I first tried putting my 3-D animated characters in a game environment almost 5 years ago, it was challenging getting the shadows right, but in the end most people think I put my characters directly in the environment instead of filming them separately in a 'green screen' environment!
HansPro
Wowee! I knew this day MUST come. As a videomaker, I tackled a short story of a small piece of firewood that fall off a truck carrying firewood, and promptly bursts into flames, hopping along the road chasing after the truck to try and be reunited with the rest of his firewood ‘buddies’. All done in realtime with real fire, no CGI. The only thing that was a major hassle was the time the story was unfolding. At sunset. Thus I only had ±25 minutes of shooting every night! Driving to and fro to the roadside set every late afternoon took a few weeks! Some days it was raining, so no footage. Recently saw ‘Love, Death + Robots’. Amazing detail on these artificial actors... so no more real actors to pay massive fees to feed their ridiculous egos! PS: We need to replace the stupid animation voiceovers using real people’s voices. Why can’t they create an artificial voice for the artificial person/ animal? One example of irritation was Shrek with Eddie Murphy’s image non-stop in your mind’s eye instead of a donkey! Aaargh! Go away!