Exclusive KFTV Virtual Production Report

KFTV VIRTUAL PRODUCTION REPORT

Sponsored by leading virtual production company and studio, LEDunit

This comprehensive report looks at how the production landscape is being revolutionised by virtual production and studios.

As the number of VP studios grow, especially across the EMEA region (see our profile list below), we talk to leading virtual production figures about how the technology is proving cost effective, sustainable and useful throughout the entire filming process. Read below...

Plus, we have case studies of leading film and TV project detailing how it works in practice; insights from location managers about how the tech is affecting their work; a Q&A with the CEO of LEDunit; and we take a look into the future of virtual production. Enjoy!...

Transforming the production landscape

Virtual Production (VP) is an evolution of established film and TV production methods with as many benefits as there are potential obstacles. With VP stages, or Volumes, shooting up all over the world filmmakers with experience of its tools are keen to share both pluses and pitfalls in order to keep virtual production on track.

“VP means a lot of things to a lot of people; it’s a very broad umbrella,” says Callum MacMillan, virtual production supervisor at London-based Dimension Studio. “At its core is a realtime games engine and the representation of 3D worlds on a giant LED wall to create the illusion of a real environment.”

In this sense VP is an extension of classic filmmaking techniques like front and rear projection or green screen in which actors are filmed against artificial backdrops for cost efficiency and greater control.

“It is the next evolution of VFX,” says John Rakich, president of the Location Managers Guild International. “What is radically different with virtual production is the introduction of an interactive light source that enables the cinematographer to capture final frames in-camera.”

Disney+ Original series The Mandalorian, for example, featured a central character with a reflective suit able to be photographed with light cast from surrounding LED panels.

VP also needs to be understood as a set of tools and processes that extend beyond the stage environment. Applications range from previz and remote scouting to character creation based on performance capture.  Assets such as models, characters and 3D environments must be completely camera-ready before production starts.  

“It takes a process that might have happened in post and moves it into preproduction,” says Steve Jelley, joint managing director at Dimension.

“It means you can shoot VFX straight through the camera with no long post process time afterwards,” adds Craig Stiff, a leading real-time artist who creates photorealistic environments and objects for virtual production and virtual reality projects.

“Department heads are shooting what they see,” emphasises AJ Sciutto, director of virtual production at Magnopus whose credits include The Lion King. “There’s no guesswork needed to ‘fix it in post’ when you can reframe a shot on set based on what you see through the lens. This production value alone, if done right, is a key motivator.”

Moving the process to pre-production also means that visual effects teams are now working with all the other departments right from the start.

“You need to lock down your script, bring in your art department, the designer, DoP and even the AD into the initial planning process, which brings a level of coordination that I don’t think people are used to,” notes Connie Kennedy, director of the Los-Angeles based Innovation Lab at Epic Games, who’s VP credits include Avengers: Endgame and The Mandalorian. “We have to make definitive decisions so that all the different elements can be composited in real time. It takes getting used to.”

“You’ll have teams communicating with each other who might not have done before,” adds Lisa Gray, executive producer at UK-based Bild Studios. “The challenge for the industry is finding agreed standard ways of working and communicating in the VP space.”

The sustainable, logistical and cost benefits

VFX houses Moving Picture Company and Industrial Light Magic laid much of the ground work to make Disney’s The Jungle Book, The Lion King and The Mandalorian. Now virtual production stages have multiplied across the world.

“The pandemic accelerated demand,” says Sciutto. “There was a lot of motivation to get back to work safely without sending hundreds of crew on location.”

Even before Covid, VP was heading into the mainstream. That’s because Hollywood may be reaching the limits of efficiency using traditional production methodology and why the predictability of the virtual environment is of value.

“You can send a two-person crew on location to scan and digitise backgrounds for filming in an LED volume,” says Sciutto. “All the variables of shooting on location such as weather conditions or where craft services will actually be can be controlled.”

Not having to set up a huge base camp or pay for travel translates to direct cost savings but indirect savings mount up too.

“Simply not unloading and repacking kit each day saves time,” says Max Pavlov, CEO & Co-founder of Latvia-based virtual production and studio outfit LEDunit. “You can shoot at golden hour all day long if you want. There’s no need to stop and wait for perfect light or fret about paying overtime.”

READ OUR FULL INTERVIEW WITH LEDUNIT CEO MAX PAVLOV HERE

Location based reshoots, which can account for 20% (or more) of the final production cost of high budget films, calculates Deloitte, can be eliminated with careful pre-planning.

This should also result in better working conditions for crew. “There’s no need to work long and anti-social hours and risk burn out,” says Sciutto.

VFX costs on a high-budget sci-fi or fantasy film can be as high as 20% of the total film budget but shooting against an LED wall significantly reduces overheads for compositing and rotoscoping. That’s because VP flips the traditional linear process of making movies on its head.

“With traditional VFX, if you want to make a change at any point you have to go all way back to that point and take the shot all the way through the pipeline again,” explains Ed Thomas, head of VP, Dimension. “With games engines we can put all the individual aspects of a linear pipe into one environment to co-exist at the same time. Potentially you have production design, lighting, layout modelling and animation all happening at the same time in one environment.”

This pooling of resources, “allows for greater creative freedom through more rapid iteration and feedback by being able to see the results prior to principal photography,” says MacMillan.

What content works best

VP stages have found a natural home for filmmakers wanting to bring fantasy story worlds to the screen but the technology is equally applicable “from a spaceship to a pack shot,” says DoP and camera operator Richard Dunton, whose credits include Apple TV+ series Ted Lasso (dailies).

“We have a permanent volume at our new Bild Studios [near Heathrow Airport], which is used by a variety of projects, each with their own story requirements and challenges,” notes Gray. “One of our first productions was a BBC documentary series with David Attenborough, which was exciting seeing how the doc guys adapted to VP. But we’ve also had scripted series coming to do car chase scenes and TV commercials too.”

Ryan Beagan, vice president of virtual production at Warner Bros. which has its own V Stage at the Leavesden site, says it opens up opportunities to shoot in areas which are difficult to travel to or simply downtown in a city where permits would be cost prohibitive. “Instead, you can use photogrammetry and Lidar scans to rebuild those locations in a studio. Other locations might let you scan, but not film in – a cathedral, for example.”

Scenes of conversations in moving cars, walk and talks and other ‘process shots’ - “things you would spend a lot of money to shoot on location can be shot on a VP stage at just the same quality without the need to send the whole crew,” says Sciutto.

A key advantage is the benefit it gives to actors able to key their performance to locations they can see rather than to invisible creatures or environments requiring greater leaps of the imagination.

“The ability for actors to play within true to reality environments is the real focus of virtual production. We are recording performances not the background,” says Philipp Klausing, executive producer and managing director at Germany-based production outfit Dark Ways, which owns the virtual production stage Dark Bay in Studio Babelsberg where Netflix series 1899 has been shooting. 

READ OUR CASE STUDY OF NETFLIX SERIES 1899

VP is not a silver bullet

But for all its advantages VP does not make creative or commercial sense for every production.

When director Denis Villeneuve invited The Mandalorian cinematographer Grieg Fraser to photograph Dune it was not to tap Fraser’s virtually unique VP expertise but the gritty realism he brought to films like Zero Dark Thirty. Both Fraser and Villeneuve wanted audiences to feel the heat and sand of a desert by shooting in the middle east.

“VP is not a silver bullet,” says Sciutto. “There are a lot of people who are trying to force things into a VP workflow that they shouldn’t. There is absolutely a tactility to locations you don’t get inside of a volume.”

Rakich advises that a volume works best when there’s a physical boundary to the scene but less well if you want to portray crowds of people. Photographing LED screens in close up can result in problematic visual artefacts which is why shots in a volume tend to have their background’s blurred and shot with a shallow depth of field.

“If your scene is set in a room as tight as a bunker it makes no sense in VP,” says Pavlov. “The camera would be so close you would probably see the pixel pitch (of the screen). You are better off building it.”

Bafta and Oscar-winning creative studio Framestore’s head of episodic, James Whitlam, advises that a dialogue heavy drama to be filmed in tight close ups would be too expensive on a virtual stage but that “a volume is perfect for recurring scenes across multiple seasons where you can achieve economies of scale.”

Shooting in a volume is not an all or nothing either.  Stages composed of modular and portable LED panels can be built to spec. “A scene set on board a train for example might only need enough panels to cover the windows to give your interior’s realistic reflections,” says Chris Hunt, CFO at LED specialist Brompton Technology.

“Virtual production is more a tool in the shed,” insists Beagan. “Depending on the project, you might still need to go to a location or a sound stage, but it’s a great opportunity to use it where it’s most applicable.”

Even The Mandalorian and 1899 make use of regular sound stages alongside a volume. This can be to accelerate schedules (by shooting different scenes simultaneously) or for aesthetic reasons. “Shooting an entire show in a volume can look too sterile” says Sciutto.

Physical builds and props on the volume stage need to merge convincingly with the CG backdrop, putting additional pressure on production design.

“The whole LED image in camera should transition seamlessly between floor, ceiling and wall of the Volume,” says Sciutto. “Most LEDs are at 90-degree angles to the stage and require some painted clean up in post.”

Warner Bros’ V Stage, which has been housing HBO's House of the Dragon series, a prequel to Game of Thrones, was designed with this in mind. “Based on the shot we are able to lower the LED ceiling and tilt it toward the camera at an angle so the image can be captured with the best colour quality (like a TV, LEDs will display the best colour when viewed head on) and so that the pixel join is seamless and looks infinite,” Beagan explains.

“The hardest thing to shoot is just a person and a screen,” he adds. “Placing physical sets or objects like a car, an obelisk or columns that repeats seamlessly into the wall works really well because the camera and your eye can fix on content that is lit in the real world.”

Physical set and virtual world blending

Nor are volume stages necessarily cheap. Depending on the project scope it can require additional labour to create VFX.

“The cost of using local realtime animators is not comparable to the outsourced prices you get from sending VFX to vendors around the world (to places like India),” says Sciutto.

Another big cost are LED panels, originally built for the digital outdoor signage industry and currently expensive rental items.

“Everything is bespoke and there’s no assembly line methodology for putting together a volume stage,” Sciutto says.

Moreover, for all the incredible photoreal quality of games engines like Unreal, this software is not capable yet of rendering some computer graphics in the highest quality. Elements like human background characters, rainforest foliage and waterfall simulations are picked out as troublesome.

“There are many interlocking parts to VP which can create cascading problems if one element is misjudged or overlooked,” says Erik Weaver, director of virtual & adaptive production at the Entertainment Technology Centre in Los Angeles. “For example, the idea that the LED screens themselves can illuminate the actors and set is broadly true but masks a maze of complications and decisions that the cinematographer and gaffer have to master.” 

Understanding the decision-making process

Technology aside, the biggest impact of virtual production is the shake-up it means to conventional filming. In theory, with the advantage of seeing finished pixels through the lens, the DOP should be more confident that their creative decisions on colour and light will translate to the final picture. For that to happen, there are new production roles and lines of decision making which need to be agreed and understood.

The Virtual Art Department (VAD) for instance, led by the VAD supervisor will include leads for creating lighting and assets within the virtual world. Their work needs to interface with a gaffer and the production designer (who works across both real and virtual sets). The ultimate responsibility for the image composition rests with the DP and director.

“Filmmakers as well as animators and crew from the games engine world both need to understand what the boundaries are,” says Klausing. “Basically, we need a ‘bible’ for delivery of content and who talks to whom.”

“There is a well-established hierarchy on a normal set. You plug in virtual production and this changes,” agrees Wilding. “On 1899 we got into a flow where we were not treading on each other’s toes but we’d have to do ten of these jobs to really know whether this is a template that works.”

Various bodies are is working to agree job descriptions and pay rates. A new lexicon for creating in virtual production is being developed by organizations including Netflix, SMPTE, EBU, VES, and MovieLabs.

Education required

Another SMPTE initiative aims to create “a set of best practices and standards,” says Katie Cole, gaming & virtual production evangelist. “At its core it is a library that will store any type of digital assets.”

That’s becoming increasingly urgent in order to slash the time in pre-production needed to build CG assets, particularly for producers needing to convince investors to finance the production.

“I am looking for an independent unified library of location-based data and a unified technology standard that will enable producers working anywhere to pick and choose assets and download them, with some adjustment, ready to play in any volume,” Klausing says.

There is also the need for skills training and education, so that there’s a continued flow of talent (from all backgrounds) working in the field, but also an exchange of knowledge and experience using the VP tech as it continues to adapt and improve to meet the demands of productions. “Much of the talent pool in the Unreal Engine [a real-time 3D creation tool] world have not been exposed to a movie set before; equally the film crew need to get their heads round the VP tech,” says Chris Ferriter, president of Halon, a visual effects studio and leader in virtual production.

Testing the technology seems to be one useful solution. “We have set up labs in London and Los Angeles where we create boot camps emulating different production scenarios, so people can learn,” said Kennedy. “We throw problems at them and get them to fix them quickly, which is what happens in reality on set. They need to be able to troubleshoot in an environment they’re not used to.”

Beyond that there are numerous education initiatives to get experienced filmmakers and graduates alike hands-on with VP stages.

“Just as when the industry went from shooting film to digital, some directors and DoPs need convincing [about VP],” says Pavlov. “But when they try it, they understand how easy it is.”

Paul Franklin, VFX producer-turned-director is a convert. “Once in a digital world your creativity is infinitely malleable. For me, virtual production is a way of taking what I would have done as a VFX artist and bringing it live onto the set.”

The Future of Virtual Production The Mandalorian. Credit Francois Duhamel VP is evolving at pace to make content creation within volumes more accessible to filmmakers. Workflow frictions will be smoothed with practice and cinematographers will gain more experience of lighting with LED walls. Technology components will require greater standardisation so that productions can get up and running quicker on stages which are less bespoke. LED technology will receive most attention. The current generation of panels were built for the advertising sector – and it shows.  “Colour temperature is a big issue,” Klausing says. “The spectrum that the display can emit is limited to RGB and not as rich as we’d like it to be.”  Sciutto points to the “hugely reflective” black plastic substrates between each pixel. “In a film environment you lose a lot of the contrast in your image because the LEDs are reflecting the light. We need denser pixel arrays or darker nano black substrates.”  The ability of games engines to render images in realtime can hit a buffer as more digitally detailed image quality is processed. It can mean that the background animation is unable to run at the same frame rate as the cameras. A solution for this is being provided by Mo-Sys to offload the rendering directly into the cloud from set.   “Mo-Sys Neartime will allow you to render Unreal scenes in higher quality while maintaining real-time frame rates,” says Grieve. “It can remove Moire (visual interference) completely – that’s a big issue in LED volumes.”  As this approach continues to advance, we’ll move towards more realistic rendering in real-time. “This can power videos screens that can turn a movie set into the Holodeck,” says Sciutto. “Having put a film crew inside the world of their story, we can soon put the audiences inside the films as well.”  Klausing further points to the benefit of VP to local language producers. “Without a big budget they have normally been prevented from travelling and restricted to making local shows in the local market. With VP they can create local language content that is located anywhere.”  Netflix is investing €500 million on German-language titles from Germany, Austria, and Switzerland over the next couple of years and has committed to house multiple series at Dark Bay.   “I envision that Germany can become a European leader in virtual production,” Netflix’s director of International Originals, Rachel Eggebeen, said recently.

 

READ: THE IMPACT OF VIRTUAL PRODUCTION ON LOCATION MANAGERS 

READ: CASE STUDY OF HOW MI6 SPY THRILLER FILM FIREWORKS TRANSITIONED FROM PLAY TO VIRTUAL PRODUCTION

 

PROFILES OF CURRENT AND FORTHCOMING VIRTUAL STUDIOS ACROSS THE EMEA REGION BELOW Winnerish Film Studios from Film Studio Pictures and Stage 50 Most of these studios will work with a choice of core software: Unreal Engine, Unity, Notch, Blender.  LEDunit Based: Riga, Lativa Facilities / Services: 32m x 5m Curved 180° Wall; 100m² Trapezoid Ceiling; Mo-Sys camera tracking; smoke extraction system; lights, grip, stage flats; make-up, wardrobe; production rooms. A second VP facility is planned for launch in Greece in 2022.  Credits: What I Was Doing While You Were Breeding (Disney+ pilot); commercials for agencies Propaganda, DDB Riga HEADQ Based: Amstelveen, Netherlands Facilities / Services: 130 m2 in-house stage with a 5.2 m high acoustically treated ceiling. A soft lit 180 degree green infinity wall; camera tracking system; lights, rigging, jib, camera, props.  Fireframe Studios Home | Fireframe  Based: Helsinki, Finland  Facilities / Services: 1000sq studio and production facilities including 18.5x6m LED wall, 12x5m LED ceiling plus moveable walls (all ROE); Nvidia and MEG Helios processing; motion capture from Optitrack; VAD  Credits: Omerta 6/12; commercials and gaming event productions Warner Bros. Studios Leavesden Based: Hertfordshire Facilities / Services: V Stage is a 7,100-sq.-ft. wraparound virtual production environment, using a matrix of more than 2,600 LED panels; motorised ceiling offering an additional 5,544 sq. ft. of LED panels with eight sections that work independently of one another, lifting and tilting to provide a new level of creative scope. Also on site: 19 sound stages, 150 acre clear-horizon backlot, on-site set lighting and rigging, and picture and sound post production.  Credits: HBO's House of The Dragon  RD Studios Based: West London, opening April 2022 Facilities / Services: includes five sound stages and 45,000 sq ft studio space, plans for a virtual production stage to open later in 2022.  80six Based: Slough, England Facilities / Services: Rental company with dual-stage facility; variety of indoor/outdoor modular and curved LED panels from ROE and infiLED; disguise and Hippotizer servers; Mo-Sys camera tracking  Credits: Commercials/corporate (UBS featuring Lewis Hamilton; Ryobi); VP test shoot Gunslinger  PRG Studio-xR Based: Paris, France; Hamburg, Germany; LA  Facilities / Services: Paris: 15 x 6 metre curved LED backdrop and 400m2 proscenium; LED ROE Black Onyx 2 / Brompton; disguise media server; Tracked Technocrane); Hamburg: 22m x 4m LED background; ceiling LED; mobile 1m x 4m LED walls; LED floor. All LED surfaces driven by Brompton processors; disguise servers; camera tracking and motion capture; camera light and grip rental.  Credits: Commercials (Orange); corporate (Omnicom); live streamed gameshow (Teammates)  Dark Bay  Based: Berlin, Germany Facilities / Services: 180.5-feet-circumference and 23-feet-high LED wall (total shooting area of 450 sq ft; modular LED ceiling; Revolving Stage to allow filming of real sets from all angles (carries up to 25 tons and can be rotated 360° in just three minutes); rain rig for rain effects; camera tracking systems VICON Motion Systems and TrackMen;  Credits:  Netflix drama 1899   Virtual Display Services  Based:   La Plaine Saint-Denis, France Facilities / Services: XR LED volume 8x8x4m; virtual set design; in-camera VFX; object and camera tracking; colour and lens calibration Credits: Soprano – live stream for Warner Music; La Soirée Extraordinaire (music show for M6) Pixomondo Based: Toronto, Canada; Stuttgart  Facilities / Services: Claimed to be the world’s largest LED volume facility ; includes VAD, virtual location scouting, previz, volumetric capture, 3D camera builds, and LED volumes with real-time environments and camera tracking.    Credits:  Star Trek: Discovery, Star Trek: Strange New Worlds Orca Studios Based: Gran Canaria, Barcelona, Madrid Facilities / Services - 6,500 sq. ft LED soundstage, previz, postviz, vfx and concept art  Credits: Paradise Hills, Down A Dark Hall, The Kill Team   Magnopus  Based: Los Angeles; St Albans, UK  Facilities / Services: End-to-end VP services (concept, VAD, virtual blocking, virtual cinematography, techviz, stage integration, stage operations, postviz, consulting)  Credits:  The Lion King, The Mandalorian  Winnersh Film Studios Based: London Facilities / Services: Opening 2022 in partnership between Stage Fifty and Fin Studio Pictures this 84m x 7m screen is claimed to be the world’s largest fully encapsulated (ceiling) volume. Can be raised 15m allowing sets to be built below. Its 210-degree circumference has side screens to wraparound to 360. Features Brompton processing and infiLED technology; disguise servers; mocap/camera tracking from Target3D; integrated motion control rolling road and turntable system.  Onset Facilities  Based: Wales, London, LA, Vancouver  Facilities / Services:  Hire on-set solutions for virtual production, VFX pre-vis and LED in-camera VFX; On-Set lens encoding service; time-coded camera tracking data export with lens distortion maps; solutions for shooting VP on green screen, LED, and outside on-location; Ncam hire, camera tracking with trained operators  Credits: Voltarol (commercial/ Hogarth Worldwide); El Eternauta (Netflix series); StormCloud (NBCU)  Bild Studios / MARS Volume     Based: London   Facilities / Services: Visual engineering and creative design; Mo-Sys tracking solutions, disguise server hardware, display technologies and AV infrastructure.  Promo for Rina Sawayama's track 'Lucid'; Le Col x McLaren campaign shoot; MTV European Music Awards  Prox & Reverie   Based: Doncaster, UK  Facilities / Services: VP facility called ‘Forge’ - a multi-reality production environment capable of pre-recording content or streaming motion-capture, volumetric and realtime performances.  Credits: Glimmer & Symphony (VP storytelling project); Micro World Exploration R&D   Dimension    Based: London and Newcastle  Facilities / Services: Partners with 80Six for VP production rental and bespoke project design and build; Pre-Viz; volumetric video capture; digital human creation; mobile XR studio ‘Polymotion Stage’; MRMC motion control; adjacent studio space supports green screen and stage builds  Credits: Fireworks (short film); Jeff Wayne's The War of the Worlds: The Immersive Experience;  Rebellion Film Studios  Based: Didcot, UK  Facilities / Services: LED wall, camera tracking, green screen, game engine visualisation, realtime rendering. Mocap and full performance capture facilities from Rebellion division Audiomotion including more than 90 Vicon cameras, head-mounted cameras, virtual cameras and full data processing capabilities. Adjacent 50,000 sq ft of space over 6 stages.   Credits: Percival (fantasy short); Ready Player One; The One and Only Ivan  PYTCH Studios  Based: Bristol, UK Facilities / Services: The space is a recent addition to one of three specialist LED wall-based film and broadcast studios that form part of the PYTCH Campus at Skyline Park, Brislington. Final Pixel Creative Studio   Based: London, UK Facilities / Services: Provides creative consulting, virtual production content (including bespoke photoreal environments) and virtual production stages and management.