Storage Pod 4.0: Direct Wire Drives

For the first time since the original Storage Pod, Backblaze is announcing a completely redesigned approach with the introduction of the first “direct wire” Storage Pod. This new Storage Pod performs four times faster, is simpler to assemble, and delivers our lowest cost per gigabyte of data storage yet. And, once again, it’s open source.

The Original Storage Pod

In order to provide unlimited online backup for $5 per month when Backblaze first started in 2007, we needed to figure out the least expensive way to “attach a hard drive to the internet.” This meant cost-efficiently attaching as many drives as possible to a single motherboard. We tried USB hubs, daisy-chaining Firewire, and various other approaches. In the end, we found one that worked: port multiplier backplanes.

We used these port multipliers to design a system with nine five-drive network attached storage (NAS) backplanes that connected via three SATA cards to the motherboard. The results were incredibly dense storage, currently 180TB in a 4U rack, from off the shelf consumer commodity parts. This design has served us through Storage Pod versions 1.0, 2.0, and 3.0, and today stores 100 petabytes of customer data in the Backblaze cloud.

However, the port multiplier backplanes had three key issues:

1. They were one of the least reliable hardware components,
2. When they had an issue, they affected five drives at once, and
3. They were not completely a commodity part, thus making them somewhat difficult to buy (especially for someone building a single Storage Pod on their own.)

The New Storage Pod Design

Fast forward nearly five years as we introduce Storage Pod 4.0 and a completely new way of attaching 45 drives to the motherboard. The five-drive backplanes are gone, replaced by individual direct wire SATA/Power connectors. In concert with this, we replaced the three four-port SATA cards with two high performance 40-port HighPoint Rocket 750 cards delivering SATA 3 throughput. Simpler to build and maintain, and seriously fast.

The New Storage Cost—$0.051 per Gigabyte

When we designed Storage Pods 2.0 and 3.0 our goal was to reduce the total cost per gigabyte to store data. Pod 2.0 was great, but when the Thailand Drive Crisis pushed up hard drive prices, we struggled to lower the cost of Storage Pod 3.0. With Storage Pod 4.0, we are pleased to announce it delivers 180TB of storage for just $0.051 per gigabyte—our lowest cost per gigabyte to date.

What’s New?

Here’s a video highlighting some of the new features in Storage Pod 4.0.

We summarized all of the changes below and as always, we’ve included a parts list at the end of the post.

Screaming Fast SATA Cards:

Storage Pod 4.0 uses two HighPoint Rocket 750 cards replacing the previous three SATA cards. Each Rocket 750 supports 40 directly connected SATA device channels via ten 6 Gbps Mini-SAS ports. The Rocket 750 card connects to the motherboard via an 8x PCIe 2.0 connection that supports eight parallel “lanes” to the motherboard. Currently, we’re seeing a four to five times performance improvement in throughput when using this set up versus Storage Pod 3.0.

• Backblaze has never had a problem with data throughput speed based on how we use a Storage Pod in production, but the increased throughput does dramatically reduce the amount of time it takes to sync the RAID arrays and burn in the system. Typically, it takes

five to six days to do these activities

when we use 4TB disk drives. With Storage Pod 4.0, this drops down to one to two days, so we can get 4.0 Pods into service much quicker.

Direct Wire Hard Drive Connections:

In previous Storage Pods, there were nine backplanes, each supporting five hard drives. There were nine SATA cables from the SATA cards to the backplanes and the backplane multiplexed to five different drives. With Storage Pod 4.0, each of the 45 drives has its own direct wire connector that provides power and a SATA connection for each drive. Each SATA connection goes directly back to the SATA card, the Rocket 750. Fifteen of the connectors are affixed to one of three cross supports in the chassis.

The direct wire connectors have multiple benefits:

• They are fundamentally simpler in design than the five-drive backplanes.
• By replacing the five-drive backplanes, all of the set up and management of the 45 drives is done on the motherboard or in the SATA cards.
• The direct wire connectors are less likely to break or burn out than the previous backplanes.
• When a five-drive backplane went down, we had five drives go offline. Now if a connector goes down, only one drive will go offline.

New SATA cables:

• Each of the Rocket 750 Cards has 10 ports with Mini-SAS connectors. The new SATA cables start with a Mini-SAS connector and fan out to four individual cables each of which terminates in the direct wire connector.

Here is a link

showing the mapping between the Mini-SAS ports on each Rocket 750 card to the individual direct wire connectors. The new cables are designed to provide both SATA and power to each hard drive. Power is provided via Molex connectors attached to a wiring harness that is connected to, and part of, the Power Supply wiring harness.

One Power Supply:

• Previous

Storage Pod

versions had two power supplies, which totaled nearly 25% of the cost of the Storage Pod chassis (not including the drives). The two power supplies were not for failover purposes, we just needed the power they created. For Storage Pod 4.0, there is only one power supply to power everything. It generates enough power to run all 45 drives, the motherboard, and all other components. Having one power supply saves on the cost and complexity, reduces the weight, and increases reliability of Storage Pod 4.0.

Thinner Chassis and New Side Rails

: One of the byproducts of replacing the five-drive backplanes is the reduction of the width of the chassis by 1/2 inch from 17 5/8 to 17 1/8. This allows us to use side rails for mounting Storage Pods in our cabinets. The previous L-bracket rails were known to be sharp in all the wrong places and more than once drew blood from our operations staff. For them, this is the most important feature of Storage Pod 4.0.

Faster CPU:

We’ve upgrade the CPU to an i3-3240 (Ivy Bridge) from an i3-2100 (Sandy Bridge) CPU. This makes processing faster and when combined with the Rocket 750 cards provided the boost in throughput of four to five times we’ve experienced. The i3-2120 and the i3-3220 have also been tested in this environment and seem to work fine, but given they are currently either the same price or a little more expensive as the i3-3240, we are currently using that CPU.

Boot Drive:

The boot drive was moved from the side to the rear of the chassis. This was done to allow for mounting holes for the new side rails on each side of the chassis. This also makes the cabling a bit easier. The new location can support either one or two 2.5 inch boot drives.

More Storage?

Storage Pod 4.0 uses 45 4TB hard drives for a storage capacity of 180TB, the same as Storage Pod 3.0. Why didn’t we use 5TB or even the 6TB drives in Storage Pod 4.0? In short, these higher capacity drives are not yet widely available and are not cost effective for Backblaze at the moment.

For example, as of March 17, 2014 an HGST 6TB internal hard drive listed on Amazon was $745.67 or about $0.124 per gigabyte with an availability of five drives.

5TB drives are also hard to find, with only LaCie’s external 5TB drive showing up on Newegg on March 17, 2014. The list price for this external drive is $449.99, making the cost about $0.09 per gigabyte, although that’s the cost of the entire unit, not just the hard drive.

Currently, 4TB internal hard drives that we use are readily available for $160.00 or less in many places, that’s about $0.04 per gigabyte.

While we’re on the topic, here are the 4TB hard drives we currently use in our Storage Pods. We have tested these extensively and are satisfied with their performance.

• HGST (formerly Hitachi): HDS5C4040ALE630
• Seagate: ST4000DM000

Although we like the HGST drives, as of March 1^st, the availability of this HGST model is becoming limited as its looks like they are being EOL’d. We currently have the Seagate 4TB drives installed in about 50 Storage Pods and they are performing well. We expect to try out the Western Digital WD40EFX (RED) drives in the near future.

Over the next few months we expect the price of the 5TB and 6TB drives to decrease. At some point we’ll buy 50 drives (45 plus spares) and install them in a Storage Pod to see how they perform in our environment. As we’ve seen previously, some drives do better than others in our Storage Pods. If we were able to use the 6TB drives, for example, we’d have 270TB of storage in a 4U rack. As always, we’ll keep you posted.

Buying a Storage Pod

If you are interested in buying a Storage Pod we recommend you talk to the folks at 45 Drives, a division of Protocase. They are very capable of building you a Backblaze inspired Storage Pod that can be customized if desired. They also can sell you a Storinator, their 180TB storage solution. The Storinator is similar to Storage Pod 4.0 in many ways, and is readily available for $5,403 for the non-redundant version and $6,888 for the redundant version. This does not include the price of the drives, tax, or shipping. While the 45 Drives system does not include hard drives, they test each system they build with drives before they ship the unit to you. Then it’s up to you to buy and install the hard drives. Remember that while the 45 Drives products are inspired by Backblaze, we do not build, test, or warrant them. Since we don’t sell the Storage Pods and don’t make money off any ancillary services, we don’t provide support for people who buy them from 45 Drives or other manufacturers.

Assembling Your Own Storage Pod

For those of you who want to assemble your own Storage Pod there is a parts list at the end of this post. You can order the 4U metal box from 45 Drives and start from there. Their 4U metal box includes the 3 metal “cross supports” to mount the direct wire connectors (see the parts list) that you can also purchase from 45 Drives. Once you have the metal box and the direct wire connectors, the other parts can be sourced as noted in the parts list or you can add in your own motherboard, memory, SATA cards, etc. to meet your needs. If you go this route, here is a wiki page, where you can learn more about the assembly process. Backblaze does not provide support or warranty to anyone who wishes to use the design to build or assemble a Storage Pod.

The Price of Storage Pod 4.0

In truth, there are actually three different prices for a 180TB Backblaze Storage Pod 4.0:

1. The price Backblaze contractually pays to build about 25 Storage Pods a month.
2. The price you would pay to purchase a Backblaze inspired Storage Pod or a Storinator from 45 Drives.
3. The price you would pay to purchase the parts and build your own Storage Pod.

In our previous Storage Pod blog posts we published the Backblaze prices, but people soon learned that they had to pay more for many of the parts, sometimes a lot more. We buy parts in bulk and as such get better pricing on many of the parts. So the real cost of building your own Backblaze Storage Pod was higher than expected. So to be clear, here are the approximate costs for a 4.0 Storage Pod with 180TB of storage, i.e. forty-five 4TB hard drives:

• Backblaze: $9,305
• Make Your Own: $10,587
• 45 Drives: $12,603

The Backblaze price of Storage Pod 4.0 of $9,305 for the complete 180TB system is about 13% less than the cost of Storage Pod 3.0 at $10,700 for an 180TB system. Of course, these prices are not guaranteed as things like hard drive shortages can change prices pretty quickly.

The Cost per Gigabyte: Onward and Downward

A Storage Pod costs Backblaze $9,305 for 180TB of storage which translates into $0.0517 per gigabyte. This is our lowest price per gigabyte ever, as you can see below:

Over the next few months, we’ll continue to look for ways to decrease the various costs and with the expected increase in storage capacities such as 5TB and 6TB drives, we’re sure the cost will drop below $0.05 per gigabyte.

Appendix A: Parts List

This is the parts list for Storage Pod 4.0, not including the 45 hard drives. The prices listed are from the sources noted and are as of the date of the publication of this blog post. Taxes and shipping are not included. You may find better prices with a little more research. Good luck.

Notes:

• Screws and other small parts are generally purchased in packages (e.g. 10 or 100), but the price list assumes you will only pay for the quantity you actually need.