Filmmaking is entering a new era, powered by the technology that's made video games some of the most compelling entertainment in recent history. It's not green-screen CGI as we know it, either – it's the creation of whole digitally projected sets.
Disney's The Mandalorian is the latest Star Wars spinoff to find massive cultural success, and one of the key challenges for this production has been figuring out how to achieve Star Wars-level visual effects and locations on a streaming series budget. Luckily, timing lined up beautifully with the development of some freaky new production techniques that have revolutionized the filming process.
Epic locations and backgrounds are crucial to a show like this, but location shooting is expensive, tricky and highly dependent on the weather and time of day. So Lucasfilm's Industrial Light & Magic VFX division decided to go with CG sets. Green screen shooting, though, can be a pain. Lighting and camera movements need to be tracked, then used to generate the final composited shots – another expensive and time-consuming process that also hinders the ability of actors, directors and film crew members to have a good idea of exactly what they're creating in real time.
Enter North Carolina's Epic Games, creators of the ubiquitous Fortnite game – better known to the older among us as the guys who made the equally impactful Unreal back in 1998. But the games themselves are only a part of the story; the Unreal Engine that powers them has become the most successful engine in history, and its ability to create 3D scenery, objects, lighting, atmospheric effects and architecture has seen it become a go-to graphics engine behind not only a slew of games from various developers, but for Virtual Reality creations and now filmmaking as well.
In a system that first debuted in 2019, Epic has created a warehouse-scale production studio that allows filmmakers to shoot live inside environments created in the Unreal Engine. It works by projecting the 3D digital environment onto giant LED screen walls and ceilings in a studio, and it can switch between generic views that allow an actor to feel part of a scene, and views that change the digital projection to match the perspective, focal length and focus position of a camera.
That means you can move the camera around as much as you like during shooting, be it handheld or on a jib or dolly, and the backgrounds will behave like you're shooting on location. But, of course, it's better than location, because you can set the time of day, weather conditions and lighting exactly how you like them and not have to touch anything up in post. You're capturing it all in camera.
What's more, you can see exactly what you're capturing in camera, no compositing required, and if you don't like where a mountain or a tree or a moon or a house is sitting, you can use the power of the Unreal Engine to drag and drop them anywhere else you want. Everything, including lighting, shadows and atmospheric effects like fog, will change immediately to accommodate, and that means reflections in shiny objects on set will be dynamic and perfect as well.
The lighting alone is a big deal. The digital world literally lights the physical space. If you move the sun in the background, or even decide to switch between day and sunset and night, you don't need to compensate for that by sending gaffers running about to change lighting gear. The lighting on the actors will match the lighting of the backgrounds, because the LED backgrounds are the lighting. And if you want to tweak the lighting at all, you can create virtual light sources of any color virtually and off-camera.
You can see the process in this excellent video from the Unreal team below.
For The Mandalorian, ILM built a cylindrical CG projection set like this, with a 75-foot (22.8-m) diameter and a 21-foot-high (6.5-m) ceiling, and the benefits immediately became obvious. Switching locations from Iceland to Scotland to entirely imaginary worlds no longer required a bunch of plane tickets and a massive logistical effort. Actors found themselves able to move, and gesture and point as if they were really standing on location. Directors found themselves able to work on dawn scenes all day, dictating exactly what they wanted the scene to look like rather than being at the mercy of the natural world.
Then things started to get creative. The team was so enamored of the incredible flexibility and speed of the system that they began shooting interior scenes in it as well as outside environments. Moving vehicle shots became a breeze. They began building physical set elements inside the virtual world, taking advantage of the dynamic lighting and backgrounds. They'd build half the Mandalorian's ship on set, so the actors could interact with it, and have the back half generated digitally. Watching the final product, you'd never know.
The upshot is a faster, leaner production – but also one that can get super creative with locations, lighting and environments; just what you want for a Star Wars piece. The Mandalorian looks and feels amazing, and it seems clear this technology will make an enormous impact on the coming generation of TV and streaming video content.
What about movies? Not so much. Many, if not most, blockbuster movies are screened in 3D these days. This Unreal Engine 4 technology can only generate a digital background for one camera at a time, so binocular shooting can't be supported. Hollywood will still need its green screens and giant battalions of CG effects artists if it wants to work in 3D.
So, we might be headed for a situation where 2D video and streaming content starts looking vastly more interesting and creative than the movies, while doing it a ton cheaper and faster. A lot of the best writing and acting talent is already moving across to small-screen productions, and this kind of technology could prove another telling blow to the world of cinema.
Enjoy a video from The Mandalorian's production team below.
Source: Industrial Light & Magic
