Not too long ago, hardware storage vendors held an iron grip on what kinds of storage underpinned your creative, film, and broadcast workflows. This storage took many complex forms — RAIDs, JBODs, SANs, NAS systems, tape robots, and more. All of it was expensive, deeply complex, and carried fat vendor margins and high support costs.
But when you’re considering storage in today’s technology environment — whether it’s cloud, on-site storage, or a USB stick — the guiding principle in choosing storage for your creative production should simply be to choose the storage that best fits each workflow step.
Production Storage Maxim: Choose the storage that best fits each workflow step
Doing your best creative work is what builds your customer base, boosts your reputation, and earns you revenue and royalties. So any time sunk into legacy storage solutions, wrestling with complexity, unneeded production steps, refereeing competing vendors, and overpaying for, well, everything, just gets in the way of what you really want to do, create.
The right answer for your specific production needs is a function of the size of your production team and the complexity of your operating environment. Whatever that answer is, it should be as frictionless an environment as possible that helps you get your work done more efficiently and gives you the most flexibility.
An independent filmmaker can follow this production storage evaluation process for each stage of their workflow and decide to make do with a small deskside RAID system for primary production storage, and depend on the cloud for everything else.
A large, global production team will probably need multiple SANs in each production office and a complex series of cloud and dedicated playout applications and systems. If your environment falls somewhere between those two extremes, then your ideal solution mix does as well.
The traditional content production workflow is thought of as a linear process. Content is ingested as raw camera files pulled into a shared work-in-process storage for editors, the final cut is then delivered to the client, and when the project is finished all files are saved off to an archive.
Simplified Production Workflow Steps
Let’s look at what the storage requirements and needs are for each of the common steps in a production workflow and where cloud can add value. Along the way, we’ll call out concrete examples of cloud capabilities at each stage with B2 cloud storage.
The Ingest Stage
Media copied in the ingest phase typically needs to get off of camera carts and flash drives as quickly and safely as possible and transported to the editing environment. Since those camera carts need to be used again for the next shot, pressure to get files copied over quickly (but safely) is intense.
Any time that critical content exists only in one place is dangerous. At this stage, lost or corrupted files mean a reshoot, which may not be practical or even possible.
Storage Needs for Ingest
Storage at the ingest stage can be very rudimentary and is often satisfied by just copying files from camera carts to an external drive, then to another drive as a safety, or by putting a RAID system on a crash cart on-set. Every team tends to come up with a different solution.
Where Cloud Can Add Value to Ingest
But even if your data wranglers aren’t ready to give up external hard drives here, one way cloud can help in the ingest stage is to help combine your ingest and archive for safety steps.
Instead of carrying carts from the shoot location to the production environment and copying them over to production storage, you could immediately start uploading content via the internet to your cloud storage, simultaneously copying over those files safely, and making them available to your entire team immediately.
When you restructure your workflow like this, you’ll get better than RAID-level protection for your content in the cloud. And by checking content into your archive first, your asset manager tools can immediately start processing those files by adding tags and generating lighter weight proxies. As soon as the files hit cloud storage, your entire team can start working on them. They can immediately begin tagging and reviewing files, and even mark edit points before handing off to editors, thereby speeding up production dramatically.
Some creatives have hit a roadblock in trying to take advantage of the cloud. Data transfer has historically been gated by the available upload bandwidth at your given location, but our customers have solved this in some interesting ways.
Producers, editors, and reporters are finding that even cellular 4G internet connections make it feasible to immediately start uploading raw shots to their cloud storage. Others make it routine to stop off at a data center or affiliate with excellent upload speeds on their way in from the field.
Either way, even novice shooters and freelancers can safely get content into your system quickly in a system that can be as simple as an upload bucket in your B2 account and making sure that your media or project manager tools are configured to watch those upload points.
Cloud Capability Example — Use a Backblaze Fireball to Rapidly Ingest Content
Backblaze offers a Rapid Ingest Service to help get large amounts of your content into your Backblaze account quickly. Backblaze ships you a 70TB storage system that you connect to your network and copy content to. When the system is shipped back to Backblaze, it is quickly moved directly into your B2 account, dramatically reducing ingest times.
Cloud Capability Example — Share Files Directly From Cloud
An example of navigating to a file-review bucket in the B2 web interface to copy the direct sharing link to send to a reviewer
In addition to the archive on ingest technique, many customers share files for approval review or dailies directly from their Backblaze B2 account’s web interface.
If your B2 bucket for finished files is public, you can get a direct share link from the Backblaze account management website and simply send that to your customer, thereby eliminating a copy step.
You can even snapshot a folder of your content in B2, and have Backblaze ship it directly to your customer.
The Work-In-Process Stage
Work-in-process or primary production storage is the main storage used to support collaborative editing and production of content. The bulk of what’s thought of as collaborative editing happens in this stage.
For simplicity we’re combining several steps under the umbrella of work-in-process such as craft editing, voiceover, sound, ADR, special effects, and even color grading and finish etc. under a far simpler work-in-process step.
As audio, color grading and SFX steps get more complex, they sometimes need to be broken out into separate, extremely high performance storage such as more exotic (and expensive) flash-based storage that then feeds the result back to WIP storage.
Work-in-Process Stage Storage Needs
Storage performance requirements in this stage are extremely hard to meet, demanding the ability to serve multiple editors, each pulling multiple, extremely large streams of video files as they edit raw shots into a complex, visual story. Meeting this requirement usually requires either equipment intensive SAN, or a NAS that scales to eye-watering size and price.
Many production environments have gotten in the habit of keeping older projects and media assets on the shared production environment alongside current production files, knowing that if those files are needed they can be retrieved quickly. But this also means that production storage fills up quickly, and it’s tempting to let more and more users not involved in primary production have access to those files as well, both of which can slow down production storage and creation of your content.
Having to make a rush purchase to expand or add to your SAN is not fun, especially in the middle of a project, so regularly moving any files not needed for current production to your content archive is a great strategy to keep your production storage as light and small as possible so that it can last over several seasons.
Where Cloud Can Add Value to Work-in-Process
By regularly moving content from your production storage you keep it light, fast, and simpler to manage. But that content still needs to be readily available. Cloud is an excellent choice here as content is both immediately available and stored on highly resilient object storage. In effect, you’re lightening the burden on your primary storage, and using cloud as an always ready, expanding store for all of your content. We’ll explore this concept more in the archive stage.
The Deliver Stage
The deliver stage, where your finished work is handed off to your customer, varies depending on what type of creative you are. Broadcast customers will almost always need dedicated playout server appliances, and others will simply copy files to where they’re needed by downstream customers, or upstream to a parent organization for distribution. But, at some level, we all have to deliver our work when it’s done.
Deliver Stage Storage Needs
Files for delivery should be moved off of your primary production storage and delivered in a separate workflow available to dedicated workflow or playout tools. Whatever the workflow, this storage needs to be extremely reliable and available for your customers whenever it is needed.
Where Cloud Can Add Value to Deliver
Whether content delivery in your workflow is met by copying files to a playout server or giving a finished file to a customer, cloud can help cut down on the number of steps to get the content to its final destination while giving you extreme reliability.
Cloud Capability Example — Serve Time-Limited Links to Content
Many customers use the Backblaze B2 API to add expiration limits that can last from seconds to a week to shared links:
An example of using the B2 command-line tool to generate time-expiring tokens for content sharing and delivery
If your team is comfortable writing scripts to automate your workflow, this can be a powerful way to directly share files simply and quickly with tools provided by Backblaze.
For more information see this B2 Article: Get Download Authorization
Cloud Capability Example — Move Content Directly to Your Delivery and Distribution Servers
Serving your content to a wide audience via your website, content channel, or app is an increasingly popular way to deliver content. And thanks to our recent Cloudflare agreement, you can now move content from your B2 storage over to Cloudflare’s content delivery network at zero transfer cost for your content application or website.For more information see this B2 article: How to Allow Cloudflare to Fetch Backblaze B2 Content
The Archive Stage
At last, we come to the archive stage of content creation, traditionally thought of as the end of the traditional content creation chain, the source of the most frustration for creatives, and the hardest storage to size properly.
Traditionally, when a project or season of a show is finished, all of the files used to create the content are moved off of expensive primary production storage and stored on separate, highly reliable storage in case they are needed again.
Archive Stage Storage Needs
Archive storage needs to be a safe repository for all of the content that you’ve created. It should scale well at a sustainable price, and make all archived content available immediately when requested by your users and workflow tools like asset managers.
Tape was often chosen to store these archive files because it was cheaper than disk-based storage and offered good reliability. But choosing tape required a large investment in specialized tape systems, tape media, and the associated support contracts and maintenance.
Tape based archiving strategies usually rely on compressing content as it’s written to tape to hit the advertised storage capacity of tape media. But video content is already stored in a compressed container, so compressing those files as they’re written and retrieved from tape offers no advantage and only slows the process down.
Here we find the chief drawback of tape based content archives for many customers: the time required to retrieve content from those tape systems. As the pace of production has increased, many customers find they can no longer wait for tape systems to return archive sets or unarchive files.
Where Cloud Can Add Value to Archive
The archive stage is where cloud has the most impact on your entire workflow. The benefits of cloud itself are familiar: the ability to scale up or down instantly as your needs change, paying only for the storage you actually use, extremely high object storage file reliability, and availability anywhere there is a network connection.
Creating The Cloud Content Backplane
Having all of your content immediately available to your production storage and your asset management systems is emerging as the killer feature of cloud for production environments. By adding cloud, your content production goes from a linear process to a highly active one where content can freely check in and out of all of your other workflow steps as you’re producing content.
By shifting your content archives to cloud like Backblaze B2, you are creating, in effect, a cloud content backplane that supports your entire content creation and delivery process with these new capabilities:
- New productions now have access to every file you might possibly need without waiting, letting you explore more creative choices
- A single, authoritative content repository backing all of your creative production lets you phase out other storage and the associated management headaches and expense
- You can now serve and deliver files directly from your cloud-based content archive with no impact on production storage
- Having content in a single place means that your workflow tools like asset managers work better. You can find files across your entire content store instantly, and even archive or move files from your production storage to your cloud content archive automatically
The content not needed on your work-in-process storage is both highly protected and immediately available wherever you need it. Your entire workflow can get much simpler with fewer steps, and you can phase out storage you no longer need on-site.
Above all, you’ll have fewer steps between you and creating great content, and you’ll be able to explore new creative options faster while shifting to a pay-as-you-use-it model for all of your content storage.
• • •
In part two, Cloud-based Tools and AI Make Workflows More Powerful and Increase Content Value, we’ll explore the ways your new cloud-delivered content archive backplane can dramatically improve how you create, deliver, and monetize content with other cloud-based technologies in the age of cloud.
