On February 11th, 2016, Adobe released an update to Creative Cloud (V 3.5.0.206) which, as they put it, “…in some scenarios the application may incorrectly remove files with user writeable permissions from the system root directory.” You can find their release notes here. This meant that if you were an Adobe Creative Cloud customer on Mac with auto-update turned on (or happened to download that version), as soon as you signed in to Creative Cloud, files from folders within your root directory could have been removed. We typically saw this occur to the user’s top-most (when alphabetically sorted) hidden folder, or to a folder that had a space at the front of its name (a semi-common practice to force a folder to always appear at the top of the list).
That update of Adobe Creative Cloud has since been pulled, and version 3.5.1.209 is available and is supposed to offer a fix.
What To Do:
If you were not running Adobe Creative Cloud on the Mac or did not update to Adobe Creative Cloud V3.5.0.206, you have nothing to worry about. If you did, and you received the Backblaze bzvol pop-up alert, you’re in luck because the file that was deleted has been re-created by Backblaze. Backblaze backups were never in jeopardy during this event.
If you were not running Backblaze or did not receive this alert, Adobe Creative Cloud may have deleted a file or folder on your computer. It’s difficult to tell exactly which data may have been removed, but you can open the root directory on your Mac and try to look towards the top for any folders that are empty. This would have occurred to only one folder (that we know of) so the top-most hidden folder or the first folder with a space as the first character would have been affected. If you do see empty folders and have a local backup, or another backup system, you may be able to restore that data from them. If not, you may want to contact Adobe for any help on this issue.
Why Was Backblaze Involved:
Backblaze keeps a hidden folder called .bzvol at the top of each drive we see connected to your system. It helps us keep track of your hard drives and keep them backed up. You can read our CTO Brian’s description of it and why we placed it there in this reddit thread. In a lot of cases, when an Adobe Creative Cloud customer hit the above bug, the first hidden folder on their main hard drive’s root folder was indeed that .bzvol. Fortunately for us and why we were able to identify this issue so quickly, is that we have a pop-up that fires whenever we detect the file .bzvol missing.
Normally, we get reports of this error very rarely. Suddenly we started seeing a rapid increase in Support tickets and Tweets related to the .bzvol pop-up. We quickly opened an internal investigation into the issue.
What Backblaze Found:
February 10th:
Wednesday night we started getting support tickets relating to the .bzvol file being removed from computers. Normally, the pop-up sends people to this (which we have since edited to highlight this current issue): bzvol webpage. The problem was, the folks on Mac kept reporting that our fix did not work and that they kept getting the error. Our support team contacted our lead Mac developer for help trying to troubleshoot and figure out what was causing this surge.
February 11th:
We continued to get a lot of Tweets, like Rob’s. People hit the .bzvol error repeatedly and our standard fix wasn’t helping. At this point, Support was receiving hundreds of tickets and chat requests (much higher than our normal ticket totals) and we were seeing a lot of Tweets coming in as well. The weird part was, none of our machines were affected, which meant it wasn’t a Mac issue across the board.
At about 3 p.m., we created an FAQ with a few instructions on how to solve our original .bzvol issues. This would temporarily fix it for some users, but many kept having this issue appear after they rebooted their machines.
On Thursday, about mid-day we caught a break. Our Head Designer, Casey, got the .bzvol error message. He told the Support team, who in turn told our lead Mac developer that we had a live use-case in the office. After a couple of hours, our lead Mac developer posted the following message in our Support team’s Slack channel:
And we were off to the races. We circled back with a lot of the Twitter folks (like Rob and Tony) asking if they used Adobe’s Creative Cloud, and they all responded in the affirmative. Support also started asking in chats and we quickly found that yes, all of these were Adobe users.
We contacted Adobe’s support and opened a case; actually we opened two separate cases. We also continued working on a work-around on our end while trying to find what was actually occurring and how we could stop it. We also opened this thread on Adobe’s forums.
This is also when we started Tweeting about the bug and trying to get people’s attention and explaining that their backups may be affected. At this point, we did not know that the Adobe Creative Cloud bug would also affect non-Backblaze customers. Below are a series of Tweets:
ATTENTION:
Adobe Cloud latest updated 3.5.0.206 causes Backblaze’s .bzvol to be deleted. Yes, it’s weird.To fix:https://t.co/jwJvDNVp8t
— Backblaze (@backblaze) February 12, 2016
For those affected, here’s a visual aid: https://t.co/33Eo7Boemz
Here is the fix -> https://t.co/jwJvDNVp8t
— Backblaze (@backblaze) February 12, 2016
More testing revealed that it wasn’t just the .bzvol folder that could have it’s contents deleted by Adobe, but any folder in the Mac root directory that was alphabetically “higher” on the list:
Worse than we thought @Adobe @creativecloud v. 3.5.0.206 update alpha sorts hidden files on main dir, nukes the top one. Reproducibly.
— Backblaze (@backblaze) February 12, 2016
Here’s a video of the Adobe bug, note the directory name, so far reproducible on Mac -> https://t.co/GEtlH8RWQn https://t.co/Ex9QUN225D
— Backblaze (@backblaze) February 12, 2016
If you had hidden folders alphabetically before .bzvol & updated Creative Cloud, check for data. If it's not there, restore from Backblaze.
— Backblaze (@backblaze) February 12, 2016
At this point we updated our help article: “bzvol FAQ” with a workaround. We created a “sacrificial folder” in the root directory which the bug would eat every time someone signed in to their Creative Cloud (even if they were still using the buggy version). We did not yet know that it could have also potentially affected customers with a space in front of their folder names (which is more common than hidden folders). We also created videos showing this happening:
February 12th:
By this point, many folks in the office were trying to help debug this problem, including our CTO. We kept testing and realized that this was occurring upon sign-in to Creative Cloud and that folders with spaces at the front of them on the root directory were also susceptible:
At about 10 a.m., we were contacted by Ars Technica, and I explained what was going on, what we had found, and who it was affecting. Since the situation was fluid, we exchanged a lot of emails and a quick call to keep Dan Goodin abreast of what we were finding.
At 12 p.m., I received emails from Adobe’s PR team, who wanted to make sure we were on the same page with how we were addressing the issue. They told me at this time that they were pulling that update and working on a fix for it.
At that time we published our blog post and sent an email to all our customers explaining that if they used Adobe Creative Cloud they might have been subject to data loss. We also let them know that Adobe had pulled the update.
Adobe also posted to their blog explaining that it was an issue affecting some folks.
As a backup company, we take data loss seriously, and when we tried to ask who exactly may have been susceptible, they did not have much information. However, we assume that anyone on a Mac that updated or was auto-updated to the affected version of Creative Cloud, and signed in, may have lost data.
We continued to field questions throughout the day and updated our help article with more concrete information as it became available.
February 13th:
There was still some confusion out in the field and in some of the articles whether or not this was strictly an issue for Backblaze users. So we ran the experiment of signing in to Adobe’s Creative Cloud without having Backblaze installed on the computer, and determined the issue could affect anyone, not just Backblaze users:
We updated our help articles to reflect that anyone on Mac may have been affected and updated our communications to reflect that.
This is *NOT* a Backblaze issue, it occurs w/o Backblaze installed -> https://t.co/XG271cweru
If ACC updated on Mac, check root folders.
— Backblaze (@backblaze) February 13, 2016
We urged potentially affected folks to check the folders towards the top of their root directory for empty folders.
By mid-afternoon, Adobe had started testing their fix and began rolling it out that night.
February 14th:
Adobe updated their blog with a link to their fix (version 3.5.1.209) and contacted us letting us know that folks should update. We’ve disseminated that information.
TL/DR Recap:
- Adobe had a buggy version of Creative Cloud (3.5.0.206) that came out on Wednesday which deleted some data inside root folders for Mac users.
- Backblaze acted as an early alert canary because we trigger a pop-up when our files are deleted from customer machines in order preserve backup integrity.
- Backblaze spent a lot of time debugging the issue and told Adobe and the public about it once we realized what was going on and who it was affecting.
- Adobe fixed the issue on Sunday with version 3.5.1.209. All users of Adobe Creative Cloud can now update their software.
- Backblaze customers who were affected can follow the instructions here: bzvol FAQ and restore their files from their Backblaze backup.
- Non-Backblaze customers who may have been affected can check other backup methods for any data that may have been removed. If you have any Adobe-related questions, please contact Adobe Support.
Many Backblaze personnel, especially Support, Engineering, and Marketing all spent uncounted hours tracking down, debugging, creating a workaround, and keeping our customers up to date on the issue. It has been a long few days.
Happy Valentine’s Day.
