Yesterday (October 24, 2013) six cabinets in the Backblaze data center lost all electricity at the same moment (10:58 a.m.). No customer data was lost, but during the repair of the electrical circuits, the Pods containing some customer data were offline for the next few hours until 9:42 p.m. when the final Pod was brought back up and online.
These six cabinets are made up of 54 Backblaze Storage Pods containing a total of 2,430 hard drives and approximately 9.7PB of storage, which is approximately one-eighth of the Backblaze fleet of Pods.
New data from all customers continued to be safely backed up onto the rest of the Backblaze Pods the whole time due to automatic software fallbacks built into the Backblaze system.
However, at 3 a.m. (October 25, 2013) a side effect of having so many Pods offline triggered some customer backups to pause temporarily. We have made adjustments and are allowing more and more customer backups to continue. Our goal is to have 100% of Backblaze customers fully working and backing up by 3 p.m. today.
We will continue to update this blog post with information as it develops. Thank you for your patience while we resolve this issue.
Update, 10/25/13 12:53 p.m. PST: Maintenance is almost complete. 95% of customers should now be fully online and able to upload data.
Update 2, 10/25/13 1:11 p.m. PST: 100% of customers are now back online and can upload data. We will continue to investigate the cause of this outage.
Update 3: All systems are normal, and Pods are online. The cause of the outage was 12 independent circuit breakers being tripped simultaneously, which affected a small percentage of our users. No customer backup data was lost. We are working with the data center to audit their power infrastructure.
