Hawkins Heroes

Netflix Originals—series, movies and other properties exclusive to or created specifically for distribution on the eponymous streaming service—number in the hundreds to thousands, depending on how you count. Some are made by production companies with deep ties to the streamer, and many more are licensed or contracted from other creators.

While Netflix and its partners filled the service’s catalogs with hours of original material, the technology behind visual effects has developed tremendously. One of the biggest innovations in filmmaking in this era has been “virtual production”, where entire soundstages are enclosed with LED video walls projecting visual effects in realtime without green-screen compositing. Entire sets are often digital creations, modeled in 3D and stored on hard drives instead of in the backlot.

Taking a step beyond streaming video content, Netflix has its eyes on gaming, metaversal experiences and other “transmedia” applications to attract and keep viewers engaged with their properties. Games based on films—and the inverse—are not a new concept, though recreating production designs in game art has never been easy. Having a vast reservoir of digital props, sets and characters at their fingertips would appear to be a huge time- and money-saver. Model and texture once, and use everywhere. Right?

In practice, 3D assets created for virtual production and other VFX are no more organized or usable than a storage unit filled with polyurethane boulders and alien masks. Because so many independent production companies work on Originals, all with their own effects contractors, models are spread out on thousands of portable hard drives, servers, cloud storage and freelancers’ laptops.

What would it take to convince production partners to see the potential future their digital creations could have?

An audit of digital assets from some very popular Netflix Original properties effectively narrowed down our focus to the precious few series and films that had their files organized well enough to even consider using. Together, they amounted to a handful of very detailed and realistic virtual production sets made in Unreal Engine, and a few key animated characters modeled in great detail with Maya.

Our own vault, at ustwo, included a wealth of experience working with Peloton to create a fitness game for their Bike called Lanebreak. Even as a strictly speculative venture, illustrating transmedia applications of these VFX assets with a plausible partnership between two huge and beloved brands generated powerful interest among our clients.

Inarguably the most widely loved Netflix Original series, Stranger Things features a group of suburban kids living out their wildest Dungeons and Dragons fantasy: fighting monsters, saving the world, and—most crucially—riding bicycles. A natural fit for a Peloton-based fitness game.

Available assets from the show included a street scene from the show’s shadow realm, complete with burned-out cars and telephone poles wrapped with slimy tentacles. Creatures like the Mind Flayer, Demodogs, and even the main villain Vecna were rigged for animation.

Designing a game around an exercise bike has some particular challenges—as input devices go, there are certainly more portable options. As we’d like to be able to work on the app from our desks (and home offices), and produce something our clients could use without assistance, we opted to develop for the iPad Air with M1 processor. Its portability, relatively performant system-on-chip, native touch controls, and great support from both Unity and Unreal Engine made it the best option for us.

A big part of the game loop revolves around moderating or intensifying exertion. To simulate that effect on the iPad, we paired the device with Sony’s DualSense controllers. By changing the resistance of the L2 and R2 analog triggers, we could show how increased resistance and cadence affected the total “output power” of the simulated bike. And while maybe not as vigorous a workout as the Peloton, you did feel it in the fingers the next day!

Of course, for ease of use during testing and maximum portability, we also built in some some simple touch controls to precisely set the output values.

Now, about those VFX assets. Game and interaction design is a crucial element of making the project easy to understand and socialize, but ultimately is a side quest to investigating and optimizing asset pipelines to turn the studios’ archives into usable game assets for branded digital experiences.

Following typical game asset optimization techniques—decimating meshes, redrawing topology to animate with fewer polygons, baking normal maps from fine detail, turning huge tiled UDIMs into smaller texture atlases—and studying the performance impact at each stage gave us data to confidently recommend a reasonable asset conversion pipeline.

To gather that data, I designed a series of tests simulating the rendering demands of our game and wrote a Unity script that let us run the scenarios on the iPad and stream debug data to the Unity and Xcode profilers.