In September 2009, Backblaze introduced and open sourced our Storage Pod 1.0 design. Storage Pod 1.0 was the result of an iterative yet informal design process where ideas were designed, built, tested, and evaluated in a matter weeks and sometimes days. Rapid iteration was key, but over time, with each major Storage Pod release, rapid iteration got less rapid. The new Storage Pod designs were good, but the process of creating them was slowing down. We had to get our design mojo back. That’s where Backblaze Labs comes in.
A Little History
Storage Pod 1.0 was preceded by multiple design iterations—some good, some bad. From USB-mass-storage banks to wooden Storage Pods, different ideas were tried. In that “loosey-goosey” environment, concepts were tried, kept, discarded, or catalogued in rapid succession.
Tim, our VP of Engineering, was a one-man shop working with a prototyping firm to progress through the iterations. At the time we only needed to build two to three Pods each month so the design and production processes could be grouped together to optimize resources. The process was simple: design it, build it, place it in production, and repeat. If the design sucked, we could work around it. As Backblaze grew, the number of Storage Pods needed each month grew 10-fold. If a design didn’t work, we had to work around 30 useless Storage Pods—an expensive and time-consuming proposition. In addition, trying to make a given design “production ready” lengthened the design process from a few weeks to multiple months—the design process had lost its mojo.
Enter Backblaze Labs
In marketing-speak, Backblaze Labs enables Backblaze to provide highly reliable, high volume data storage at the lowest cost in the industry. Instead, let’s just say that Backblaze Labs is all about trying out cool ideas. For the bigger cool ideas, we needed to have a way to quickly gather, capture, test, and evaluate ideas over and over again without bothering our “normal” business. Let’s see how this worked as we worked on cool designs for our next Storage Pod.
Divide and Conquer
We started with Backblaze Labs running both the Storage Pod design and production processes. On one hand, we had a design process that was iterative and ever changing. On the other hand, we had a production process that was highly standardized so we could benefit from procurement and production efficiencies. Different goals, different ways of thinking—it didn’t work. After some more experimentation, Backblaze Labs ended up owning the design process while the production process was managed by Procurement.
Iterate to Improve
“When in doubt, iterate,” is the mantra of our design process. In the case of Storage Pods, there were a number of ideas we had on how to improve the design. For example, the location and number of fans we used. Storage Pod 4.5 has six fans, two rows of three each. Too many? Not enough? Just right? Different locations? All good questions and all things that could be tested. What we needed was a good way to track, manage, and evaluate the different iterations.
Agile, Scrum, and Sprints
When Tim was “the” design team, he could generally keep the design process in his head. As more people got involved, we spent more and more time managing the process and less time doing anything else. We needed a process that supported team iteration and incremental development. We adopted the Agile methodology and used the Scrum framework which we applied to hardware design. We’ll walk through the Agile process of how we designed the next generation Storage Pod in a future post, but for now we can say that we used the Scrum framework to organize and run a series of four-week Sprints to iterate the design.
It is unusual to use Scrum for hardware design, but here are some of the benefits we found to date:
- Rapid prototyping of designs allowed us to quickly flush out good and bad ideas.
- Having assembly and service personnel on the design team allowed us to respond to their issues during the design phase. This should lead to making assembly and maintenance easier and less expensive downstream.
- We have a defined way to produce a “production ready” Storage Pod while continuing to innovate the design.
Find the Right Partner
About a year ago, Evolve Manufacturing started working with us to assemble Storage Pods. Their assembly facilities and processes are ISO-9001 and ISO-13485 certified and they help create products serving the medical, life sciences, aerospace, and defense industries. Their facilities are located nearby and they are believers in onshoring—performing technology-based assembly work here in the US.
One of the things that impressed us was how efficient they made the Storage Pod assembly process. With that in mind, we approached them about working together to design Storage Pods using the Agile methodology. They agreed and have fully supported the ongoing Storage Pod design Scrum. Their commitment has been instrumental in turning ideas into prototypes for each new design iteration.
What Else Does Backblaze Labs Do?
Besides designing Storage Pods, Backblaze Labs has the responsibility of testing new technologies that we need to consider. One such technology is shingled magnetic recording (SMR) drives. Hard drives that use SMR technology write new data tracks that overlap part of the previously written tracks. Will SMR technology affect the way Backblaze reads, writes, and deletes data from a drive? What are their performance characteristics in the Backblaze environment? What are their failure rates? These are the types of questions Backblaze Labs needs to shed light on.
Backblaze Labs is also the place we start when investigating updates/upgrades in Storage Pod components, such as motherboards, power supplies, or SATA cards and cables. These types of projects are typically short-lived with few if any iterations.
The People of Backblaze Labs
A good question to ask is, who works in Backblaze Labs? The answer is no one and potentially anyone at Backblaze. There are currently no permanent employees in Backblaze Labs, but for a given activity, a project team is put together under the Backblaze Labs umbrella. Here are two examples:
Long Term: The current Storage Pod project consists of a people from Procurement, Operations, and multiple departments within Evolve that utilize the Scrum process for managing the project. This project will last several months and produce multiple production ready Storage Pod versions. Over the course of the project, individuals may swap in and out as needed.
Short Term: On the other hand, a project such as testing and/or qualifying a new motherboard may last only a few days or weeks. There is no need to form a Scrum team as the task is straightforward and there are only a few people involved. The expected output is a pass or fail on the qualification along with the appropriate documentation.
More Than a Box
Backblaze Labs is not a box on an org chart with a manager and a budget, it is our way of dynamically combining people, process, and mojo together to render ideas into reality. Stay tuned…
