When the director of the Cybersecurity and Infrastructure Agency calls a vulnerability “one of the most serious I’ve seen in my entire career, if not the most serious,” ears perk up.
The director was referring to the Apache Log4j vulnerability that was discovered this month. Some more colorful phrases used to describe the Log4j incident include: “a grave threat,” “a design failure of catastrophic proportions,” something that will “haunt the internet for years.”
The vulnerability proceeded to set off five-alarm fires in IT, security, and operations departments around the world. Or should have, at least. Researchers estimate at least 840,000 attacks have since been launched via the vulnerability since it was discovered. That is to say, if you’re using a software or cloud vendor that hasn’t made some kind of statement or taken corrective action, you should be asking them why not.
At Backblaze, we made the decision to take our servers temporarily offline in order to hunt down potential threats, apply the appropriate security patches, and test those patches to help prevent our systems from being compromised. This post explains why we made the decision, outlines the actions we took to meet our objective of securing customer data as well as our environment, and provides more insight into our process.
What Is the Log4j Vulnerability?
As reported by ArsTechnica, a zero-day vulnerability was discovered in the Apache Log4j logging library that enables attackers to take control of vulnerable servers. Though it may not be an immediately recognizable name, Log4j is widely used throughout the world by companies like Apple, Twitter, and Tesla as well as the game Minecraft. The library allows developers to easily log application events. The Cybersecurity & Infrastructure Security Agency (CISA) urged users to apply patches immediately to address the vulnerabilities.
Our Decision
Upon learning of the Log4j vulnerability, our team took swift action to investigate and assess available options to address the potential impacts since Log4j is leveraged widely in our environment. As part of our investigation, our internal team used a nondestructive form of the exploit to confirm our vulnerability. We also noted close to 80,000 unsuccessful Log4j exploit attempts on our sites in a 12-hour period. The level of activity, along with our success using the exploit (albeit with internal knowledge of our own systems), was very concerning to us.
Although we were not aware of any unauthorized access to our systems due to the Log4j vulnerability, out of an abundance of caution, we decided it was in our customers’ best interest to take systems offline until they could be patched. The decision to take our systems offline was not one we took lightly. However, our Incident Management Guidelines are quite clear. In a crisis where tradeoffs must be made, our descending list of priorities (all of which are very important to us) is as follows:
- Health & Safety.
- Data Integrity & Confidentiality.
- Service Availability.
- Service Performance.
Protecting customer data integrity is second only to health and safety and above service availability. That said, the decision to temporarily bring all services down was unprecedented in the 14-year history of Backblaze. This was an extraordinary case where we made a decision to take a necessary action to address an imminent risk of a vulnerability with a Common Vulnerability Scoring System (CVSS) score of 10.0—the highest possible score. We believe that we needed to take preventative steps to protect customer data by temporarily taking our services offline until the security patching process was complete.
What Actions Have We Taken?
A recap of recent actions is outlined below:
- Upon learning of the Log4j vulnerability, our Security team took immediate action to investigate.
- Based on our assessment of the potential threat, we decided to temporarily take our services offline to apply a security patch to prevent our systems from potentially being compromised.
- We announced our systems had been taken offline at 5:20 p.m. PT on December 10, 2021.*
- We announced our systems were back online and functioning normally at 3:01 a.m. PT on December 11, 2021.
- Based on our investigation, we also determined that there was no evidence of our systems being compromised or unauthorized access to customer data or files due to the Log4j vulnerability.
*We decided not to announce downtime publicly until after our systems were offline to avoid any elevation of priority to those targeting our services. Accordingly, we did not make a public announcement until after the servers were disconnected.
Was Backblaze Compromised?
We have not found any evidence of system compromise or unauthorized access to customer data or files at this time.
Next Steps
As is part of our incident response process, we always look for ways to do better and identify areas for improvement. In this case, two top priorities moving forward would be to improve how we can apply security patches faster and reduce downtime.
Thank you to our customers for your understanding as we navigated this challenging incident.
