The rain started in August and by mid-October 2011, violent floods in Thailand had damaged the factories that helped produce nearly half of the world’s hard drives. As an online backup company, Backblaze fills more than 50TB of new drives every day. To survive this crisis without raising prices or compromising service, Backblaze deployed every last employee, as well as friends and family, to acquire drives in what became known internally as “drive farming.” What follows is how we did it.
Good Morning, Your Costs Just Tripled!
Brian, our buyer, gasped, “$349 for a 3TB internal Hitachi hard drive!” How could that be? The last drive order he placed just two weeks ago on NewEgg was at $129 per drive. He re-checked the SKU three times. He found Seagate and Western Digital prices were similar. What was going on? Over the next few days, prices spiked in all of our traditional purchasing channels; everyone was quoting ridiculously high numbers. One “friend” offered us 100 drives, non-returnable, for a mere $568 each—we said no. In a matter of days, the cost of the hard drives that are core to our business had jumped over 200%, meaning the cost of providing our customers’ unlimited storage was going to skyrocket as well. We needed a plan.
We Didn’t Plan for Rain in Thailand
We started drive farming in November 2011. The reason was simple: the supply of the 3TB hard drives used in our Storage Pods had dried up—or more correctly—was under water. The tragic flooding in Thailand began in August 2011 and by early October, it had submerged houses, schools, and factories. Over 800 people died and many more were homeless and hungry, with over one million people thrown out of work. As the water receded, the human cost of the flooding was obvious and the economic impact was slowly coming into focus. In late October, it was estimated that up to 50% of the worldwide hard drive manufacturing capacity was lost or damaged. The impact of the lost capacity was immediate as hard drive prices nearly tripled overnight.
Wanted: A Really Good Idea
We huddled in the conference room. CTO Brian stood at the whiteboard listing retailers, store locations, and employee names. While we had a Storage Pod buffer in our data center that could last a few months, some industry analysts were already predicting that the drive shortage could last well over a year. Backblaze is committed to providing unlimited data backup, but with our normal channels charging usury prices for the hard drives core to our business, we needed a miracle. We got two: Costco and Best Buy. On Brian’s whiteboard, he listed every Costco and Best Buy in the San Francisco Bay Area, and then some. We would go to each location and buy as many 3TB drives as possible.
Shucking Is a Good Thing
The going rate for a 3TB external drive at Costco or Best Buy was $169. Internal drives, like those inside your computer and the ones we typically use in our Storage Pods, were at least $100 more and were becoming nearly impossible to find, having been bought by HP, Apple, and Dell for their systems. Our “choice” to use external drives meant they had to be “shucked”—a process where the internal drive is removed from its case—before being deployed into a Backblaze Storage Pod. In addition to the price of the drive, there were now extra costs to shuck the drives, extra costs to recycle the shucked parts, and once shucked we couldn’t return a bad drive, but it was still less expensive than buying internal drives even if we could find them. Problem solved; Costco here we come.
“Aaargh, There the Treasure Be”
Backblaze employees Yev and Ken stared, exhausted, into the car trunk at their treasure—52 3TB disk drives. That would be enough for two, maybe three days at the current burn rate, they thought, as they drove back to the office to unload their bounty. They hoped Damon, KC, and the other Backblaze employees would have similar success and we’d add a few more weeks’ worth of hard drives to our stockpile.
 |
 |
The Economics of Storage Pods
Backblaze provides online backup for unlimited data at $5/month. We can do this because we have implemented our own backup and data storage management software on our own Storage Pods. These Pods are built from commercially available parts such as 3TB internal hard drives from Seagate or Western Digital, motherboards from Supermicro, etc. Each Storage Pod holds 45 drives or 135TB of data. We can reliably store and retrieve data at up to 25 times lower than the cost of other services such as Amazon S3 by using our own purposed-designed cloud storage.
Did You Really, Really, Really Mean Two?
The “two drive limit” signs started appearing in retail stores in mid-November. At first we didn’t believe them, but we quickly learned otherwise. Sometimes, we talked our way into more, but we heard “two is the limit” a lot. We started doing “drive math:” two drives a day per store, times three stores per day, times five “farmers,” times seven days a week is 210 drives. That would be sufficient, but in reality it didn’t work out that way. Stores were stocking out of drives on a regular basis and we really couldn’t farm every day, but we kept at it. One Wednesday afternoon, after working all day at Backblaze, Yev circled the San Francisco Bay hitting local Costco and Best Buy stores—10 stores, 46 drives, 212 miles on his Nissan.
Family and Friends to the Rescue
In late November, Brian was banned from buying drives on Costco’s online store, and Billy was banned from purchasing them at any Costco in the Bay Area. Other Backblaze employees were also asked to leave stores empty handed. Other vendors, along with the seasonal demand from Christmas, were rapidly draining supplies. It was time to get creative, again. At an impromptu drive crisis meeting, fueled by stale pizza and good beer, we decided we needed to scale drive farming and the idea of friends and family drive farming was born. Emails and text messages were sent, phones calls were made, Facebook posts were posted, Tweets were tweeted, you name it. The call went out to friends and family: Buy hard drives and send them to Backblaze, NOW.
Simon Says, “Raise Prices”—Backblaze Doesn’t Play Simon Says
While we were drive farming, other vendors reacted more traditionally. Intel warned that the drive crisis could lower its quarterly revenue by about $1 billion, give or take $300 million. EMC notified its Velocity channel partners, “Our Q1 2012 HDD list prices will rise between 5-15% over Q4 2011 levels.” HP announced that “HP will be forced to increase the prices that we charge for certain disk drives.” Dell sent us a nicely worded email explaining that “HDD costs have increased…and Dell’s list pricing has been adjusted as a result.” So while we pondered the economics of renting a truck and driving it across the country buying drives along the way, others raised prices—we would not do that.
Drive Farming at Work
Stores throughout the rest of the country were also limiting hard drive availability, but now we had more farmers. Drives started to trickle in, two at a time. It was cheaper to buy external drives at a store in Iowa and have Yev’s dad, Boris, ship them to California than it was to buy internal drives through our normal channels. Nearly every day, the postman would deliver packages of drives from friends and family around the country. Tim would pile the boxes up on a table in the office and every week the van from our Storage Pod assembly company would pick them up. A few weeks later the Storage Pods, filled with 45 drives in each, would arrive in the data center. There, Sean and Guido would deploy the Pods to store the petabytes of data backed up by our customers.
Holiday Drive(s)
On Christmas Eve, Gleb, our CEO, stopped by a friend’s house to pick up 80 3TB drives his friend had acquired when an online site forgot to limit the quantity he could order. It had taken the FedEx guy nearly 30 minutes to unload them and carry them up to his friend’s apartment. As Gleb was lugging boxes of hard drives to his car to take back to Backblaze, he realized that the hard drives he was loading were worth more than the car he was loading them into.
More Stories
There are many great adventures from our days as drive farmers. Here’s one:
Getting Back to Normal
Drive farming continued throughout January and into mid-February of 2012. By then, we had farmed 5.5PB of data storage. Eventually smaller distributors and resellers appeared offering decent prices on internal drives—no more shucking. Throughout the crisis, we didn’t raise our prices, we continued our policy of unlimited storage, and we didn’t throttle our customers’ backup speeds. Prices have slowly decreased over the past several months as Thailand has gotten back on its feet and production shifted to other factories. We recently purchased another 2,000 internal drives to add to our stockpile. Lower prices are nice.
Thank You
We want to say thank you to our friends and family who helped us through the drive crisis: Boris, Cara, Jim, Susan, Dave, Meng, Mike, Ben, Evelyn, Katherine, Randy, Ramey, Lise, Pete, Paul, Vladik, Tasha, Hakan, Alec, Alla, Katia, Leya, Yan, Jessica, Sergey, Mary, John, Eugenia, Penny, Sergey, Rebecca, Tim, and many others. We’d also like to thank the retailers who kept the price of external drives reasonable—even Costco.
Yet, even though we at Backblaze weathered the drive challenge, it pales in comparison to the challenges faced by the people of Thailand as they continue to recover from the devastating floods that have ravaged their country. Our hearts go out to them. If you wish to help, please consider a donation through Give2Asia or the relief organization of your choice.
Epilogue
On July 25th of this year, Backblaze took $5 million in venture funding. At the same time, Costco was offering 3TB external drives for $129 about $30 less than we could get for internal drives. The limit was five drives per person. Needless to say, it was a deal we couldn’t refuse. Old habits die hard.
