There’s almost no way to quantify the impacts COVID-19 has had on the world. Personally, communally, and economically—there isn’t a part of our lives it hasn’t touched in some way. We’ve discussed how it’s affected our operations and our culture, but at the end of the day, the central focus at Backblaze is providing the best cloud storage and backup in the world—a mission that’s especially important in a time when cloud storage and data security has become more vital to day-to-day life than ever.
At the most basic level, our services and products rely on a singular building block: the hard drive. And today, we’re going to discuss how our team has ensured that, as more businesses and individuals turn to cloud storage to solve their rapidly evolving data storage and management needs, we’ve had what we need to care for the petabytes of inbound data.
We’re no strangers to navigating an external threat to business as usual. In 2011, flooding in Thailand impacted nearly 50% of the world’s hard drive manufacturing capability, limiting supply and dramatically raising hard drive prices. At the time, Backblaze was only about four years into providing its computer backup service, and we needed to find a way to keep up with storage demand without going broke. We came up with a hack that became internally known as “drive farming.”
What does it mean to farm hard drives? Well, everyone on our staff, and many of our friends and family members went out and bought every hard drive we could get our hands on, at every retail outlet nearby. It was a bit unconventional, but it worked to maintain our storage demand. We wrote the whole story about how we weathered that crisis without compromising our services in this blog post.
This year, most of us thought the eruption of the volcano Taal in the Philippines was going to be the biggest threat to the hard drive supply chain. We were wrong. Instead, we’ve been called to apply some of the resourcefulness we learned during the Thailand drive crisis to deal with the disruptions to production, manufacturing, and supply chains that COVID-19 has caused.
No, this isn’t “Drive Farming: Part II, the Drivening!” Rather, faced with an uncertain and rapidly shifting business environment, we turned to someone on our team who knew, even before 2020 began, that a global pandemic was a much more likely challenge to our operations than any volcano: our Senior Director of Supply Chain, Ariel Ellis.
Recently, Ahin (our VP of Marketing) sat down with Ariel to discuss how he has been managing our supply chain efforts within the context of these extraordinary times. The Q&A that follows has been edited for brevity (give a marketer a microphone…). It covers a wide range of topics: how business has changed since the emergence of COVID; how our supply chain strategy adjusted; and what it’s like for Ariel to do all of this while battling COVID himself.
Ahin Thomas: Wow! What a ride. Let’s start by understanding the baseline—what was considered “business as usual” in the supply chain before COVID? Can you give me a sense of our purchasing volumes of hard drives and who makes them?
Ariel Ellis: Pre-COVID we were buying hard drives on a quarterly basis and deploying around 20-30PB of data storage a month. We were doing competitive bidding between Seagate, Toshiba, and Western Digital—the only three hard drive manufacturers in the world.
AT: It doesn’t seem that long ago that 30PB in a year would have been a big deal! But are you saying things were pretty stable pre-COVID?
Ariel: Everything was relatively stable. I joined Backblaze in 2014 and pre-COVID, 2019 has probably been the most consistent in regards to the hard drive supply chain that I have seen during my tenure.
AT: Well that’s because neither of us was here in 2011 when the floods in Thailand disrupted the global hard drive supply chain! How did the industry learn from 2011 and did it help in 2020?
Ariel: The Thailand flooding caught the manufacturers, and the industry, off guard. Since then the manufacturers have become better at foreseeing disruptions and having contingency plans in place. They’ve also become more aware of how much routine cloud storage demand there is, so they are increasingly thoughtful about our kind of businesses and making sure supply is provided accordingly. It’s worth noting that the industry has also really shifted—manufacturers are no longer trying to provide high capacity hard drives for personal computers at places like Costco and Best Buy because consumers now use services like ours, instead.
AT: Interesting. How did we learn from 2011?
Ariel: We now have long term planning in place, directly communicate with the manufacturers, and spend more time thinking about durability and buffers.
I was actually brought in right after the Thailand crisis because Backblaze realized that they needed someone to specialize in building out supply strategies.
The Thailand flooding really changed the way we manage storage buffers. I run close to three to four months of already deployed, forecasted storage as a buffer. We never want to get caught without availability that could jeopardize our durability. In a crisis, this four-month buffer should provide me with enough time to come up with an alternative solution to traditional procurement methods.
Our standard four-month deployed storage buffer is designed to withstand either a sudden rise in demand (increase to our burn rate), or an unexpected shortage of drives—long enough that we can comfortably secure new materials. Lead times for enterprise hard drives are in the 90-day range, while manufacturing for our Pods is in the 120-day range. In the event of a shortage we will immediately accelerate all open orders, but to truly replenish a supply gap it takes about four months to fully catch up. It’s critical that I maintain a safety buffer large enough to ensure we never run out of storage space for both existing and new customers.
As soon as we recognized the potential risks that COVID-19 posed for hard drive manufacturing, we decided to build a cache of hard drives to last an additional six months beyond the deployed buffers. This was a measured risk because we lost some price benefits due to buying stock early, but we decided that having plenty of hard drives “in house” was worth more than any potential cost savings later in the year. This proved to be the correct strategy because manufacturers struggled for several months to meet supply and prices this year have not decreased at the typical 5% per quarter.
AT: So, in a sense, you can sacrifice dollars to help remove risk. But, you probably don’t want to pull that lever too often (or be too late to pull it, either). When did you become aware of COVID and when was it clear to you that it would have a global impact?
Ariel: As a person in charge of supply chains, I had been following COVID since it hit the media in late December. As soon as China started shutting down municipalities I recognized that this was going to have a global impact and there would be scarcities. By January most of us in the industry were starting to ask the question: How is this going to affect us? I wasn’t getting a lot of actionable feedback from any of the manufacturers, so we knew something was coming but it was very hard to figure out what to do.
AT: That’s tough—you can see it coming but can’t tell how far away it is. But seeing it in January—on a relative basis—is early. How did you get to that point?
Ariel: I’m part of the Backblaze COVID preparation team and the Business Continuity Team, which is a standing team of cross-functional leaders that are part of the overall crisis response plan—we don’t want to have to meet, but we know what to do when it happens. I also had COVID. As we were making firm business decisions on how to plan for disruptions I developed a cough, a fever, and had to take naps to make it through the day. It was brutal.
In December of 2019, we realized we had to stay ahead of decision making on sourcing hard drives. We had to be aggressive and we had to be fast. We first discussed doing long term contracts with the manufacturers to cover the next 12 months. Then, as a team, we realized that contracts weren’t an option because if shelter-in-place initiatives were rolled out across the country then we were going to lose access to the legal teams and decision makers needed to make that process work. It was during the second week of March 2020 that we decided to bypass long term contracts and do the most viable thing we could think of, which was to issue firm purchase orders. A purchase order accepted between both companies is the most certain way to stay at the front of the line and ensure hard drive stock.
We immediately committed to purchase orders for the hard drives needed to cover six months out. This was on top of our typical four-month deployment buffer and would ultimately give us about 10 months of capacity. This is a rolling six months, so since then I’ve continued to ensure we have an additional six months of capacity committed.
Issuing these purchase orders required a great deal of effort and coordination across the Business Continuity Team, and in particular with our finance team. I worked side-by-side with our chief financial officer to quickly leverage the resources needed to commit to stock outside of our normal cycles. We ordered around 40,000 hard drives rapidly, which is about 400PB of usable space (meaning after parity), or roughly $10 million worth of capital equipment. Overall, this action has proved to be smart and put us one to two weeks ahead of the curve.
AT: We’re all grateful you made it through. OK, so a couple weeks into a global pandemic, while you’ve contracted COVID-19, we increased our purchasing by an order of magnitude! How are the manufacturers performing? Are we still waiting on drives from the purchase orders we issued?
Ariel: We’ve deployed many of the drives we’ve received, and we have a solid inventory of about 20,000 drives—which equals about a couple hundred petabytes of capacity—but we’ve continued to add to the open orders and are still waiting for around 20,000 drives to finish out the year. The answer to manufacturer performance changes on a constant basis. All three manufacturers have struggled due to mandated factory shutdowns, limited transportation options, and component shortages. We consistently experience small-to-medium delays in shipments, which was somewhat expected and the reason we extended our material buffers.
AT: Is there a sense of “new normal” for the buffer? Will it return to four months?
Ariel: This is going to change my world forever. Quarterly buying and competitive bid based strategies were a calculated risk, and the current crisis has caused me to rethink risk calculation. Moving forward we are going to better distribute our demand across the three manufacturers so I stay front and center if there is ever constrained supply. We will also be assessing quarterly bidding, which while price effective, gives us limited capacity and it is somewhat short-sighted. It might be more advantageous to look at six-month, and maybe even rough, 12-month capacity plans with the manufacturers.
This year has reminded me how tentative the supply of enterprise hard drives is for a company at our scale. We rely on hard drives for our life blood and the manufacturers rely on a handful of cloud storage companies like us as the primary consumers of high-capacity storage. I will continue to develop long term supply strategies with each of the manufacturers as I plan the next few years of growth.
AT: I know we are still very much in the middle of the pandemic, but have things somewhat stabilized for your team?
Ariel: From a direct manufacturing perspective we’re just now starting to see a return to regular manufacturing. In Malaysia, Thailand, and the Philippines, there were government-imposed factory shutdowns. Those restrictions are slowly being lifted and production is returning to full capacity in steps. Assuming there is no pendulum swing back to reinfection in those areas, the factories expect to return to full capacity any day now. It’s going to take a number of months for them to work through their backlog of orders, so I would expect that by October we will see a return to routine manufacturing.
It’s interesting to point out that one of the most notable impacts of COVID to the supply chain was not loss of manufacturing, but loss of transportation. In fact, that was the first challenge we experienced—factories having 12,000 hard drives ready to ship, but they couldn’t get them on an airplane.
AT: This might be a bit apocalyptic, but what was your worst case scenario? What would have happened if you couldn’t secure drives?
Ariel: We would fully embrace our scrappy, creative spirit and I would pursue any number of secondary options. For example, deploying Dell servers, which come with hard drives, or looking for recertified hard drives, drives that were made for one of the tier one hardware manufacturers but were unused and went back to the factory to be retested and recertified. Closer to actionable would have been slowing down our growth rate. While not ideal, we could certainly pump the brakes on accelerating new customer acquisitions and growth, which would extend the existing buffers and give us breathing room.
AT: Historically, the manufacturers have a fairly consistent cycle of increased densities and trying to drive down costs. Are you seeing any trends in drive development driven by this moment or is everyone simply playing catch-up?
Ariel: It’s unclear how much this year has slowed down new technology growth as we have to assume the development labs haven’t been functioning as normal. Repercussions will become clear over the next six months, but as of now I have to assume that the push towards higher capacities, and new esoteric technologies to get to those higher capacities, has been delayed. I expect that companies are going to funnel all of their resources into current platforms, meeting pent up demand, and rebuilding their revenue base.
AT: Obviously, this is not a time for predicting the future—anyone who had been asked about 2020 in February was likely very wrong, after all—but what do you see in the next 12 months?
Ariel: I don’t think we’ve seen all of the repercussions from manufacturing bottlenecks. For example, there could be disruption to the production of the subcomponents required to make hard drives that the manufacturers have yet to experience because they have a cache of them. And whether it’s through lost business or lost potential, the hard drive manufacturers’ revenue streams are going to take a hit. We are deeply vested in seeing the hard drive manufacturers thrive so we hope they are able to continue business as usual and are excited to work with us to grow more business.
I think there will also be a further shift towards hard drive manufacturers relying on their relationship with cloud storage providers. During COVID, cloud service providers either saw no decline in business or an increase in business with people working from home and spending more time online. That is just going to accelerate the shift of hard drive production going exclusively towards large scale infrastructure instead of being dispersed amongst retail products and end-users across the planet.
Even more surprising is that our team hasn’t changed that much. Something Ariel mentioned after we finished our conversation was how, when he had to go offline due to complications from COVID, the rest of our team easily stepped in to cover him in his absence. And a lot of those folks that stepped into the breach were the same people wheeling shopping carts of hard drives out of big box stores in 2011. Sure, we manage more data and employ more people, but when it comes down to it, the same scrappy approach that got us where we are today continues to carry us into the future.
Floods, viruses, volcanoes: We’re going to have more global disruptions to our operations. But we’ve got a team that’s proven for 14 years that they can chart any uncertain waters together.
