Rodeo FX finds the flexibility of scale with Microsoft Azure HPC + AI

Creativity doesn’t just happen. It requires inspiration, commitment, the freedom to push boundaries, and the tools and resources that turn dreams into reality. Rodeo FX knows this firsthand. The Montreal-based visual effects (VFX) and animation studio uses its creative chops to add magic to some of the world’s biggest film and television productions from leading content producers like Disney, Marvel, Netflix, Amazon, and Apple. The secret is providing its artists with the space and support they need to create gripping, game-changing stories.

A key component of that support lies in Rodeo FX’s embrace of dynamic, cutting-edge technology. The studio’s drive to seek out challenges and charge into projects other studios balk at is complemented and supported by its robust and reliable digital infrastructure. So, when Rodeo FX saw a way to gain even greater reliability with added scale and compute to keep its artists’ creativity flowing with Microsoft Azure HPC + AI, it did what it always does: dive in feet-first.

The challenges of digital rendering

Achieving realistic, lifelike VFX requires significant amounts of digital heft. Artists build out wireframes for each frame of animation, which are then treated to layer upon layer of detailed additions—converting an outline of a city street, for instance, into a lush and living depiction of a late nineteenth-century New York City streetscape. Accordingly, each frame of animation consists of numerous files that then need to be rendered to compose a single finished frame. To handle the scale and compute required for rendering, most studios maintain “rendering farms,” a cluster of numerous on-premises servers dedicated to this one purpose.

With the rendering of a single frame taking anywhere from an hour to several days, and a production requiring up to 120 frames per second, this process consumes a lot of time and computing resources. The amount of productions a studio can take on, then, becomes limited by its rendering capacity. Benoit Rainville, Rodeo FX’s DevOps Lead and Systems Administrator, explains, “Our rendering farm is never idling for long—it is at 90 percent–plus usage, every day, 365 days a year.”

More HPC, less TBD

Productions can fluctuate wildly in terms of resource requirements. A single sequence consisting of numerous frames, for instance, might require one artist, or it might need up to 200—depending on complexity. Project life cycles add to this fluctuation: as a project ramps up, it demands more and more artists, who pile on more and more rendering work. If, for example, a project takes three months, likely 50 percent of the rendering will be done in the same two-week period, causing a huge spike in computing resource requirements.

To better ensure that productions get to and through rendering more immediately, Rodeo FX looked to the cloud. Because it already had a high-performance on-premises rendering farm, the studio pursued an architecture solution capable of extending that asset, adding cloud nodes to its on-premises HPC cluster for additional scale. For Rodeo FX, preserving and extending the value of its on-premises asset was key. Tony Zotta, Global Head of IT at Rodeo FX, states, “We had invested millions on our local render farm, so it just made more sense for us to use that asset and burst into the cloud as necessary.”

Rodeo FX, then, needed to develop a hybrid estate to keep itself versatile and agile, in order to take on any and all exciting new projects that came along. Jordan Soles, Rodeo FX’s VP Technology and Development, recalls, “In 2014, after we delivered Game of Thrones season four, we saw how impactful that show was on our studio as a whole. Over the summer, and before season five, we worked to build a burstable hybrid render strategy to utilize the cloud and our local render resources, which we’ve relied upon since to manage as many concurrent shows as possible. Given our history and experience in the field, it made sense for us to explore the Azure solution because it took the data management step off our plate, and simplified our ability to achieve our ultimate goal of creating the best image possible.”

The solution uses Azure ExpressRoute for its network connection, alongside Azure high-performance computing instances, to support and augment its on-premises render nodes as necessary. But what really sets Rodeo FX’s architecture apart is its deployment of Azure HPC Cache, which keeps the workflow between render nodes running in Azure and the on-premises storage environment. As Andrew McKenzie, Rodeo FX’s System Administrator, explains, “Our caching grabs the data from our main filer on-premises, and then serves it to the computing instances that are in the cloud.” The rendering farm is managed by Rodeo FX’s render wrangling team, who, McKenzie continues, “choose when and which jobs to move to the cloud, by simply signing into Azure, accessing the virtual machine scale set, and scaling up the instances themselves. It’s that simple.” The amount of render nodes the solution uses varies, depending on how much work needs to be done on the farm, but it can reach a maximum of a thousand for each of its two caching servers.

To gain the greatest efficiency in its resource usage, the solution uses two different Azure regions for its cloud rendering requirements, finding the highest availability while maintaining the lowest possible latency. Zotta recalls, “Azure HPC Cache was a real game-changer; it simplified everything. It made it easy for us to take the same rendering process we apply locally and move it to the cloud.”

Animating from anywhere

In addition to gaining greater scale and flexibility with cloud bursting, Rodeo FX also uses Azure to support its remote team members. While COVID-19 made remote work commonplace, it had an additional consequence for film and television. The cessation of filming in 2020 created a dearth of content and an ensuing spike in demand from studios once filming recommenced. Accordingly, the need to find the best VFX talent to satisfy demand meant studios had to look beyond their localities. In Rodeo FX’s case, it wanted to support not only a new studio in Toronto, but also artists working in and around Vancouver, Canada. Rodeo FX uses GPU optimized virtual machines (VMs) to provide workstations for its artists working remotely, so they can function as if they were inside the Montreal studio.

Zotta says, “With Azure Virtual Desktop, we can have our artists work from anywhere. We can recruit and hire artists from across the country and put them in a workstation that connects them directly to the studio.” Adds Rainville, “With Azure, we have the technology that enables people to work from home. That means we can attract better talent because people don’t have to leave family and friends behind.”

The value of versatility

With its cloud architecture, Rodeo FX gains greater scale, flexibility, and versatility. The ability to turn on and off capacity as necessary translates to better project and cost management. Rodeo FX can pay for cloud bursting as needed and not have to carry the costs of maintaining additional capacity on-premises. Indeed, if it were to match the scale gained by cloud bursting, Rodeo FX would need to double the size of its on-premises rendering farm—a costly investment from which it would only benefit during spike periods.

Scale aside, Rodeo FX also gains greater process simplicity and agility through its architecture. Rainville states, “We wanted to reduce the time to first pixel rendered, but we also needed a system that was easy to administrate. With Azure HPC Cache, we gained speed and optimization. We only have to press a button—it’s that easy.” The simplicity of the solution contributes to Rodeo FX’s ability to be more responsive, too. Rainville continues, “Azure  HPC + AI gives us added agility in the sense that we can get to cloud rendering much faster than we used to. We can eliminate the problems that come with a packed queue at deadline. We can absorb the wave before it gets too big.”

Best of all, the scale and flexibility that Rodeo FX has developed is giving it a leg up in terms of winning new business and competing with much larger studios. Rodeo FX’s solution provides that extra guarantee that it can find the scale and capacity it needs, when and as the studio needs it. Zotta says, “What cloud bursting with Azure HPC Cache gives us is the capacity, control, and visibility to deliver projects when we need to.” Rainville concludes, “We don’t have a massive render farm like some of the big studios, but with Azure HPC + AI, we have economies of scale, so we can bid for the biggest movies and shows and compete with the top players in the industry.”

Find out more about Rodeo FX on Twitter, Facebook, and LinkedIn.