Simplifies Storage and Processing for 70+ Million Files

Macro Storage for Microstock Images and Videos

Flash back to the early 2000s—the world wide web was booming. Web designers were using rich imagery in creative new ways. Digital cameras had reached the mass market, inspiring hobby photographers everywhere to capture high quality images with relatively inexpensive gear. However, there were few options to connect the two communities as traditional stock agencies only offered high priced photos from a limited pool of professional photographers.

‍

Duncan Enman saw an opportunity to reframe this picture. As both a photographer and a web developer, he personally understood the challenges on both sides. He explained, “I wanted to create a solution that allowed amateurs to sell their work and get a fair deal and also made quality images affordable for websites.”

‍

Enman set his sights on the nascent microstock industry, which focused on licensing large volumes of images at only a few dollars each. So in 2004, Enman launched his own microstock agency, Can Stock Photo, to offer a more competitive solution in the space.

‍

In the beginning, Can Stock Photo received a few hundred images per day. Photographers would upload their work via a browser or FTP, Enman or volunteers (usually other photographers) would verify the quality and legality of every image.

‍

At that time, Enman thought, "If we ever got 10,000 photos, that would be enough. We’d have a picture of everything and the library would be reasonably well rounded." Little did he know that he’d reach that goal within only a few months.

‍

Fast forward to today—Can Stock Photo provides over 70 million images from 40,000 photographers and artists worldwide. The company reviews 30,000 assets per day using a team of inspectors augmented by artificial intelligence tools. The company also supports stock photography programs that automatically share images with multiple agencies.

‍

‍

Physical Storage Got Really Expensive, Really Quick

Can Stock Photo started off with a virtual private server (VPS) shared hosting plan including a couple hundred megabytes of storage in a co-located facility—and ran out of space in the very first week. Whenever a new server was needed, Enman would have to sign a contract and have it provisioned; as a result, hosting costs skyrocketed.

‍

Enman recalled, “I started this dance of adding storage, running out, and adding more to keep up with growth. When we got into the gigabytes of storage, it seemed absolutely insane. And of course we’re way past that now."

Capturing Scale in the Cloud on AWS

Enman turned to Amazon Web Services in 2006, becoming one of the earliest adopters of Amazon S3 storage. Suddenly, the business could grow seamlessly with far less effort. “The cloud allows you to be so scalable,” said Enman, “and that was a great improvement for us as we do a lot of processing under the hood.”

‍

So, Can Stock Photo migrated all of its content to Amazon S3 and all image processing to AWS EC2, Amazon’s compute platform. Dynamic scaling helped to better process simultaneous uploads (sometimes hundreds of photographers uploading at once). As volume increased, Enman and team decided to free up active storage space and reduce costs by archiving some of the less frequently used content in Amazon Glacier.

‍

However, the company ran into several critical issues with its AWS infrastructure. Operating costs were a top concern for the bandwidth-intensive business and Amazon’s pricing model was complex and costly with increasing egress fees. As the business grew, Enman and team worried about vendor lock-in. If they ever wanted to leave AWS, the cost to switch providers would only grow steeper over time—the more data they accrued, the more expensive it would be to leave.

‍

In addition, the team wanted to optimize their storage setup. Glacier imposed a delay and costs associated with retrieving inherently cold files. Reprocessing an old asset to make a new thumbnail meant submitting a request to retrieve the asset, paying associated costs, waiting for retrieval, and using the asset within an allotted time. It was comparable to dealing with an LTO tape system—slow, tedious, and costly if you want your data faster.

Exposure to Backblaze Reveals a Better Solution

Three considerations were top of mind in the company’s search for a new solution: cost, reliability, and performance. The other major infrastructure as a service providers, however, did not offer a compelling fit.

‍

Enman had successfully been using the Backblaze Personal Backup service for a number of years, so he decided to take a closer look at Backblaze’s business solutions. As he learned more about Backblaze B2 Cloud Storage, he felt that it could be a suitable replacement for Amazon S3. What’s more, Backblaze’s Cloud to Cloud Migration service brought the cost of pulling data out of AWS to Backblaze B2 down to zero.

‍

“Our team already had a lot of trust in Backblaze,” said Enman. “We read the Backblaze Blog all the time; we’re always interested in the Drive Stats reports. We’ve had many internal conversations about the Backblaze findings.”

‍

So, Enman reached out to set up test accounts. “If you’re already using the Amazon S3 API, as we were, it was just a matter of changing end points within our application,” said Enman, “Everything worked the same on Backblaze B2 as it did on Amazon S3.”

‍

It took two weeks for Can Stock Photo to migrate over 600TB of data from AWS S3 to Backblaze B2. Enman recalled: “If I rewound the tape to a year ago, I’m sure I expected transitioning all of our content anywhere would be a prolonged and painful process. The reality of switching to Backblaze couldn’t have been further from that—it was so uneventful and so easy. There was no downtime at all; it was literally like flipping a light switch on the back end.”

‍

The team ran their own internal audit to ensure everything migrated over as expected and they found no issues. After launch, they ran both systems in parallel as they slowly migrated more and more services over from Amazon S3.

Running a Better Digital Darkroom on Vultr

Backblaze’s partnerships, particularly with compute provider Vultr, were icing on the cake for Can Stock Photo. With a wide range of partner integrations and incentives available, the team saw many potential benefits with switching to Backblaze partners. They evaluated several partners and chose Vultr to host their image processing application. “Vultr reminds us a lot of Backblaze,” said Enman. “They’re very responsive and keen to serve customer needs.”

‍

As new content comes in, all processing is now handled on Vultr. The system runs video compression and transcoding, saving the process files on Backblaze, and it also creates standard sizes and file formats for both videos and images. If customers want a different size, the system can resample the file at the time of purchase. Processing includes creating various marketing assets, such as thumbnails and previews, as well as watermarked and unwatermarked versions. The system also handles vector-based files, rendering them server-side.

‍

“Transitioning to Vultr has also been nice and seamless,” said Enman. “We’ve been pleased with their server specs, usage has been simple and straightforward, and performance has been really good.”

A More Nimble Profile for Future Growth

The switch to Backblaze was completely invisible to customers and content creators. Internally, however, the team has seen some important improvements. They can now generate video files for customers four times faster with Backblaze than they could using AWS. And cost savings in particular have been huge—monthly storage is 55% less with no transfer fees between Backblaze and alliance partner Vultr.

‍

Moving cold storage files from Glacier to Backblaze has also eliminated retrieval costs and removed the time and effort of managing Glacier’s complexities. All files are kept in hot storage and are easily accessible to the team when needed.

‍

Most importantly perhaps, Can Stock Photo feels less locked in to an expensive and complex solution. Enman said, “With Backblaze, our business is in a more nimble position. We can react to the market easily, and we reduced operating costs for storing and processing video while creating a better end user experience.” In particular, the company can focus more on the video space, as well as innovate new ways of delivering better experiences for content creators and customers.