With the astounding popularity of the animated feature films from Paris-based Illumination Mac Guff, it would seem the whole world already knows about the secret lives of pets and the villainous plots of Gru and his goggle-eyed minions. But few moviegoers—or even the studio’s own digital artists—would guess the inner workings of the production systems behind the stories.
And that, says Illumination Mac Guff head of technology Bruno Mahe, is by design. “Our primary objective in IT is to protect the artistic intention for every project. We do that by prioritizing support for digital artists, delivering maximum responsiveness to their workstations and providing seamless access to digital assets, regardless of the load on the underlying IT infrastructure or the location of datasets.
“Avere Systems FXT Edge filers in front of our storage systems ensure highest performance to both desktops and the studio’s 80,000-core render farm,” said Mahe. “Using Avere technology, we can maintain predictable service to artists, even as datasets are hammered with thousands and thousands of renders. Avere also serves as a gateway to our on-premises media archive, HGST object storage system, from Western Digital, which provides affordable, online access to multiple petabytes of indispensable media assets. The Avere and HGST solutions let us effectively shield studio artists from data access complexities and fluctuating rendering workloads, keeping them productive and free to focus on artistic vision.”
Illumination Mac Guff is a wholly owned subsidiary of Universal Pictures and one of the world’s leading studios for animated feature films. Founded in 2011, the company has experienced phenomenal success with such megahits as the Despicable Me films, Minions, The Secret Life of Pets, and Sing.
The studio first deployed Avere technology in 2012 and since that time has expanded its production pipeline, increased crew from 400 to some 1,000 artists during peak development, and more than doubled per-film global active data. “We also nearly doubled I/O requirements—twice, actually, in just a matter of months,” said Mahe. “Such growth comes from our commercial success, as well as the expanding expectations of the industry and movie audiences. In terms of animation and story quality, every film must be better than the last.”
Mahe says one of the studio’s biggest challenges has been keeping up with archive capacity requirements. “We keep all of our digital assets online and have traditionally used second-tier, super-dense NAS with high-availability features. In addition to feature films, we are doing more to engage the audience beyond the theater and are now generating a lot more content per film. The second a project is over, the assets need to become a reference library. This library must be online at all times for the content to be quickly found, reused and monetized. With our archive at two petabytes and growing, we needed a more economical on-premises active-archive solution for the long term.”
Today, an Avere FXT Edge filer cluster serves as a gateway to HGST’s ActiveScale object storage system at the Illumination Mac Guff data center, providing transparent NFS-to-S3 (Amazon S3-compliant interface) object storage translation for seamless access to the archived data. Some two petabytes of digital assets have already been consolidated from a variety of second-tier NAS systems, to provide reliable online access to the studio’s entire archive of finished productions. Using the Avere technology with the ActiveScale system gives Illumination Mac Guff a more cost-effective archive solution that still delivers required performance with massive scalability and high durability at petabyte scale.
Avere clusters also front-end some six petabytes of production-data capacity across two tier 1 storage clusters. Avere ensures high-performance I/O to the studio’s render farm, as well as to approximately 1,000 artist workstations.
In 2017, the studio’s Despicable Me franchise became the highest-grossing animated film franchise of all time worldwide. Mahe says that Avere provides essential enabling technology to support the growth that comes with such success. “We’re typically working on multiple films at various stages of development (and all of equal priority), as well as an array of other short movies and ancillary marketing projects that boost active data by at least 15 percent. To give you an idea of the increasing pressures on our storage infrastructure, peak I/O on the 2016 Sing production averaged 450K IOPS. Just months later on Despicable Me 3, demand peaked at 1.2M IOPS. Peak dataset size on Despicable Me 2 hit 800TB, and by Despicable Me 3 active data had jumped to nearly 2PB.
“Using the Avere cache we don’t have to recalculate as much data, we’re keeping pace with growth, and we’re able to deliver consistent performance without the peak-production slowdowns that studios would normally experience producing movies at this scale,” he said. “And, we’ve actually improved responsiveness to artist workstations. While a one-second delay in a twenty-four render is acceptable, a three-second delay at an artist workstation is never tolerable in our environment.”
Mahe says it’s always about performance—for rendering workloads, responsiveness to workstations, and active access to archived characters, bitmaps, and other digital assets for reuse. “The ActiveScale solution gives us the required combination of speed, petabyte scale, durability, and the economics of cloud in our data center. Even using sophisticated heuristics we can’t predict which assets artists will need, so the entire asset library must be online and reliably available. The on-premises archive also allows us to meet our legal obligations for dataset security.”
The Illumination Mac Guff studio boasts a strong R&D culture. “We have developed a number of in-house systems, including our ray tracer and asset management tools,” said Mahe. “The Avere gateway lets us immediately utilize the object archive without any changes to user access or workflows, while the ActiveScale system gives the company on-prem native S3 capability for the future. It’s the ideal combination for immediate value with long-term flexibility.”
“We looked at a number of gateway/object-storage systems,” said Mahe, “but no other vendors offered equivalent levels of tested interoperability, direct industry experience, and expert support. The HGST and Avere solution provides essential performance, scale, and economy, but it was the depth and strength of the corporate relationships—between Avere and HGST and with Illumination Mac Guff—that sealed the deal. This is a long-term solution, and we’re confident of long-term support, continued investment in the technologies, and vendor viability.
“At one point last year we were managing a pipeline of four feature films, and I remember thinking, ‘We can do this, we can support multiple major projects without production slowdowns.’ The Avere and HGST technologies make it possible to fully support artistic intent—we don’t have to be the IT team that restricts creativity, telling artists and directors that their visions are too expensive.”