Streaming Platform Rebuilds AVOD Infrastructure for Performant Savings

Cost-Effective Scalability Gets Top Billing on Plex

For over 22 million fans around the world, logging into Plex is like being handed the keys to a candy shop. A cornucopia of free delights awaits them on the media streaming platform, from thousands of on-demand movies and TV shows, to live TV, to podcasts, music, and even one’s own personal media collection.

‍

Plex offers a library of content from many of the biggest brands in entertainment and serves it all up through a rich, integrated user experience. The Plex dashboard also offers content discovery through friend recommendations and deep links to titles on other streaming providers, making it easy for consumers to quickly find and access that interesting show they’ve been hearing about.

‍

With an ever-growing body of content and massive distribution needs, Plex depends on cost-effective scalability. After launching its advertising-based video-on-demand (AVOD) service, the company faced staggering egress costs from its all-in-one hosting platform—around $70K per month and growing. Plex made a strategic decision to rebuild the product’s infrastructure in-house using specialized cloud products Backblaze B2 Cloud Storage and Cloudflare’s content delivery network (CDN) to optimize data storage and content delivery, respectively.

‍

As a result, Plex achieved more control over its data without needing to overhaul code thanks to the Backblaze S3 Compatible API. And reduced its egress bill to $0—all without sacrificing performance. Here’s how they did it.

Scaling AVOD, the Sequel

In 2019, the media streaming market was booming, and Plex wanted to deliver its own streaming solution as quickly as possible. The opportunity was ripe, and the ad-supported model offered another dimension to the Plex business model. In fact, some content is available only on ad-supported services rather than on paid services due to the complexities of studio agreements. 

‍

So, the company launched its AVOD service on a SaaS video hosting platform connected to Amazon S3 on the backend. The all-in-one solution provided all the components of the workflow, enabling the Plex team to take the video files they received from film and TV studios, transcode them using Zencoder, and serve them to consumers. 

‍

It worked well for a time and delivered a great user experience for fans and studio partners—until Plex began to scale the service. As the AVOD service rapidly grew, it became a core component of the Plex business overall. As such, the team wanted to expand by adding new features and tools for curators, but the hosted solution proved limiting, both in terms of cost and vendor lock-in. They decided to bring hosting in-house in order to gain flexibility and greater control.

‍

As they began rebuilding the infrastructure for their AVOD service, the team discovered a few hidden costs associated with the previous solution. One of the biggest was the high egress costs of moving content to and from S3, especially since large video files are typically broken up into smaller chunks for streaming. They also store several copies of a given title for different screen resolution and/or localization, which are broken up into smaller segments for storage as well. The team realized that they could save a huge amount by using an independent content delivery network (CDN), so they began formulating a new solution.

A Short Story: Backblaze B2 Migration

“We had been talking about moving to Backblaze B2 for a long time, just because of the cost savings on storage,” said Scott Olechowski, Co-Founder & Chief Product Officer at Plex. “But Backblaze also offered an S3 compatible API and a relationship with Cloudflare that meant no CDN egress fees, so it was almost a no-brainer for us to try and make it work.”

‍

Moving to Backblaze B2 also meant that the team didn’t have to disrupt much of their code on the backend. With the Backblaze S3 Compatible API, they could just change the storage destination to point to Backblaze, so doing an initial proof of concept test was quick and easy. They set up a test bucket on Backblaze B2 pointing to a Cloudflare CDN and added some test content, and used Grafana’s K6 load testing tool and Mux monitoring tools to load test and benchmark performance. 

‍

Once they were satisfied with performance, they added a few low-traffic production titles to see how the set-up performed with real users. User testing results showed no appreciable difference, so they made the switch. “Our plan was to just swap the DNS when we went live with everything, so that Cloudflare would just point to Backblaze instead of S3,” said Brian Robinson, Senior Software Engineer at Plex.

‍

Migrating 350+TB data over to Backblaze was faster than expected. “We were worried that we'd have to stop ingesting new content for a week,” said Robinson, “but migration to Backblaze B2 only took a weekend, and by Monday or Tuesday, we were able to start ingesting content again. The whole process was basically turnkey, thanks to the Backblaze S3 Compatible API.”

Terabytes Ready for Showtime

Although most traffic comes in during U.S. peak viewing hours, Plex sees a steady drumbeat of traffic across its global user base—to the tune of 9PB per month served out through Cloudflare, and that’s just for the video streaming part of the business.

‍

Plex offers over 50,000 on-demand titles, receiving new titles from studios every month. It all gets prepared, encoded, and stored on Backblaze B2 for delivery to end users when available in their country, and during the timeframe that Plex holds streaming rights. Plex inserts advertisements into the streaming experience, all of which flows seamlessly from its origin on Backblaze. 

‍

Even when the rights to a particular title expire, Plex continues to store that title in case the company renews the rights in the future. This means that Plex’s Backblaze B2 storage continues to expand on an ongoing basis. Currently, the company is storing 400+TB of data from 505 million files.

‍

The Plex Workflow Script 

For the Plex team, ingesting and preparing content is a straightforward process. When a studio partner sends Plex a new title, it’s typically a huge file. The team uses Zencoder to encode the file and produce several resolution variants, such as 1080p, 720p, 480p, and more. Each is an individually encoded stream that is then chunked into five, 10, or 15 second pieces of video, which are then stored in Backblaze B2. 

‍

“We handle the packaging and segmentation ourselves,” said Robinson. “So, Zencoder will create what we call our transcoded renditions, and then we have our own tool to package the HLS live streaming protocol and the DASH streaming format, and then apply the DRM before it goes to Backblaze B2.” (Those acronyms stand for HTTP live streaming (HLS), dynamic adaptive streaming over HTTP (DASH), and digital rights management (DRM), for those not familiar.)

‍

When a user chooses to view a title, their video player chooses the appropriate quality files based on bandwidth available and retrieves them via the Cloudflare CDN.

‍

‍

The Residuals: A Flexible Stack, More Control, Lower Costs

After rebuilding its AVOD infrastructure, Plex realized a number of benefits for the business. First and foremost was cost savings. “Ours is not a high margin business, so anywhere we can improve the cost profile makes a difference,” said Olechowski. “By moving to Backblaze B2 and Cloudflare, our egress fees went from $70K/month to $0/month.” Egress is free between Backblaze and Cloudflare. And the Plex business is designed to keep growing indefinitely, so that savings becomes more and more meaningful over time.

‍

For the moment, the team plans to invest that savings back into the product by focusing more on developing new features and services, bringing in content that will drive higher engagement, or enhancing their marketing efforts. “As we go deeper into this space, we’ll see greater technical requirements from the studios,” said Robinson. “We’ll want to find ways to reapply our cost savings to other areas that help us unlock better content.”

‍

The greater flexibility and control with their new stack allows the team to better integrate the AVOD service with other parts of the Plex platform in a more efficient way. It also enables the secure flow of content—a key concern for studio partners—and it delivers great performance for end users. Going forward, the Plex team is looking at other ways to use Backblaze B2, such as storing metadata, images, and other parts of the streaming experience, as well as opportunities with other integrated partners. 

‍

“Overall, our experience with Backblaze has been pretty much smooth sailing,” said Olechowski. “Everyone's been really happy, and it's a big win on our side.”