VR Training Simulator

MGM Resorts International is a hospitality and entertainment juggernaut, managing over forty properties—in Vegas and all over the world. That includes shopping centers, hotels of all kinds, restaurants, nightclubs, arenas, conference centers and, yes, casinos.

Backed by the resources of such a massive concern, MGM's innovation team worked with our immersive media team at ustwo to introduce “metaverse” technologies into the company and demonstrate their value to practical business use cases.

An open-ended project goal like the one presented to us creates a dependency on a sort of meta-project: finding purpose. Before we could get to work, we had to answer some important questions like: who should our users be—customers and guests, or folks inside the company? Should we focus on generating revenue, or creating efficiencies? Will internal stakeholders respond better to glitzy presentation or hard data? Our strategy for finding these answers included methods like the “concept equation” workshop, where participants from our internal and client teams match cards in different categories to generate ideas. Chris Marotta, our lead designer, explains the process in the video session from AWE, "Playful Tools for Serious Work" (see footnote [2]).

Once settled on our opportunity area of creating efficiency within the company's Security & Surveillance apparatus, our next step was to spend time with them on-site and understand their day-to-day work, pain points that seem unavoidable, and areas where we could add value.

Our team was given access to the inner workings of the Aria Resort & Casino, one of the most luxurious properties on the Las Vegas Strip, and embedded with their security team for a 24-hour tour of duty. Think of what a security guard stationed on a busy gaming floor deals with all day—drunk and disorderly guests, people angry about losing all their money on slot machines, people belligerent from winning too much money on slot machines, the potential for chaos from fights, fires or active shooters… all real possibilities, and we did see some of that. Something else, though, emerged as the biggest use of time by far: giving directions.

The truth is, Aria is huge—and it's not even the biggest resort in Vegas. People don't know where anything is, and they ask security guards for help: how to find their way to hotel elevators, their dinner reservations, where to catch a cab, how to cash in their winnings. All that depends on guards knowing where everything is.

MGM sends their security staff through a rigorous training academy, where they learn every inch of the place on top of conflict resolution, de-escalation, customer service, and probably jujutsu for all I know. Training takes weeks, and two of them are devoted primarily to physical orientation. On top of all that, turnover among security staff is high, so the academy is constantly busy with new recruits.

So we've finally found our niche: let's use VR to recreate the Aria and put guards-in-training in it from day one to increase the speed and effectiveness of classroom training.

From this point, two processes continued in parallel: refining the game loop, and building a faithful recreation of the resort that could be run on the Quest 2 headset. I took the lead on the digital twin, starting with processing CAD and blocking out the environment to scale.

A majority of the volume of the Aria's ground floor is devoted to gaming—rows and rows of slot machines and green felted tables. Their sheer number represented both a challenge and a relief: they needed to look convincing, but didn’t need to be precise. Despite their importance and quantity, they were ultimately the least of my concerns. Entrances to major points of interest, general material and lighting characteristics, and accurately replicating wayfinding signage were my main focus.

The environment was constructed by combining CAD data with 3D scans of key items, 360° video, and tons of photos captured on-site.

Working within a strict performance envelope on the Quest 2, and needing to update the Unity project regularly for playtesting, optimization was critical to consider from the beginning. The entire Resort model was created from 157 unique 3D assets—instanced, where possible—with a total polygon count under 60,000. Casino gaming features were considered separately, instanced with a script that randomized textures, and aggressively culled by distance and camera at runtime. This left us with more than enough overhead to also run a crowd simulation, giving life to the virtual space.

Our sprawling labyrinth of psychedelic carpeting and strobing slot machines created the same challenge in-game as it does in real life: it takes a long time to get around and is easy to get disoriented in. To manage the tedious traversal between training scenarios, we introduced a “Surveillance View” inspired by the interface that Aria staff use to select cameras in their headquarters. The new view provided quick access to key locations and gave players an overview of active scenarios that they could choose to train on.

We also provided a virtual radio over the Quest's left motion controller, which showed quick access to key information like the number of active scenarios, time remaining in the day simulation, and a mini-map with waypoints. The full bird's-eye view was always accessible through the radio—switching perspective from the guard working the floor, to the operator dispatching orders.

Not every scenario that a guard-in-training is presented with is a lost guest in search of soup dumplings. For more high-stakes interactions, we developed an enhanced conversation experience based on full-motion video with spoken dialog.

To produce these scenarios, the Resort model was brought into Unreal Engine—the simulator was written in Unity and optimized for the device there—and given a more cinematic treatment. The script for each scenario was voiced by ElevenLabs's AI text-to-speech service, which was then used to generate detailed, synchronized facial animations with Nvidia's Audio2Face tool. These animations were targeted to Epic's MetaHuman facial rig, and rendered out as high-quality clips that could be overlaid into our Quest app, thus letting us keep the simulation running well and affording realistic video content that trainees could feel the emotion behind.

Using the training academy's teaching materials, we created a handful of scripted scenarios delivered along with the pilot app. Instructors could also use a custom scripting and content management tool to add to the library. The system was tied into MGM’s employee portal to let trainees log in with their credentials to track and save progress or receive a tailored curriculum.

Using the VR training simulator, new security guards became comfortable operating in the real Resort within two weeks!

We proposed alternative uses of the digital twin, and a documented process for producing twins for more properties, in the form of a surveillance camera positioning simulator (allowing planners to preview their camera views before installing them):

Our next target was MGM's conference and events business—set to exceed casino gaming by revenue within years—which was especially interesting to me having designed many corporate events in exhibit halls and conference centers. Lots of venues provide great CAD plans of their exhibition spaces, but very few provide details on what “connective tissue” lies between them: hallways, escalators, doors that don't stay open, places to get food and rest. Those important features can often only be seen on site visits—and scheduling those can be tough.

Leveraging our digital twin, the whole team can be looped in on a virtual site visit to preview event locations and build a shared vision. Information availability can be a powerful selling point and help MGM close sales before their competitors.

Once the sale is made, a detailed web-based conference simulator can help planners avoid bottlenecks and maximize their schedules.