School District Tests Its Way to Scalability Gains and Predictable Costs

Making the Grade: Bethel School District Gets an A+ in Storage

‍Bethel School District is booming. With 23,000 students, 3,500 staff, and 30 locations, Bethel is one of the fastest growing K–12 public school districts in Washington state. Every new student, teacher, administrator, and employee increases the volume of data that flows through the organization every day, challenging Bethel’s IT team to manage it all effectively across an ever-expanding fleet of servers. And as a publicly funded entity, they have to do so within a limited and closely monitored budget.

‍

Pencils, Chalk, and Terabytes of Data

Patrick Emerick is a senior systems engineer at Bethel with deep experience on both sides of the district’s growth trajectory. Not only do his four children attend schools in the district, but he’s also responsible for the infrastructure that those schools depend on. “My team is currently supporting an average of 200 servers at any given time,” he said, “which run the district’s entire operations. It’s about 125TB of archive data in total.” 

‍
All of that data needs to be backed up, and those backups need to be stored on off-site servers for disaster recovery purposes. “Anything that a server can do,” said Emerick, “it needs to be backed up and stored somewhere. The reason we retain these backups is for ransomware and for making sure we've got a good backup that can't be altered.” To better prepare for growth, Bethel switched its disaster recovery archive from Amazon S3 to Backblaze B2 Cloud Storage, resulting in a more cost-effective storage solution at scale. Here’s how they did it.

‍

Failing Grades: Archiving on Amazon S3 Was Too Expensive

Bethel's technology infrastructure is primarily located on-premises with only a few cloud services in the mix. The IT team uses Rubrik, a data management platform, as its backup system. Rubrik takes a snapshot of each server, replicates it, and creates a backup file that is initially stored in the district’s on-site data center. “That server snapshot contains a whole grab bag of data,” said Emerick. “It could be anything from student databases to teacher communications to our whole Windows environment and automation scripts.”
‍

The team needed to store a copy of the backup file off-site in case of a cyberattack or natural disaster at their data center. So, when they implemented Rubrik several years ago, they connected it to Amazon S3 in order to archive copies of their backup files to the cloud. Over time, this proved to be an expensive setup—in the realm of thousands every month. 

‍
“We were paying through the nose for Amazon S3 just for archiving,” Emerick recalled, “and we needed to find ways to reduce costs during a very tight budget cycle.” Unfortunately, they were forced to reduce costs in a less than ideal way—by shortening their retention periods.

IT Does Their Homework on Cloud Storage

The team evaluated Amazon Glacier’s Deep Archive and Flexible Retrieval, but found that Amazon’s complex pricing structure could result in unpredictable costs. Emerick said, “Glacier sounds great on paper, but then they nickel and dime you with their various fees and tiers. It makes it hard to test scenarios and determine exact costs. I understand Amazon is a big player in the space, but their pricing structure is obscure on purpose, and that's just not ideal for us.” 

‍
Due to Glacier’s retrieval fees and warming delays, Bethel would struggle to figure out how much they’d have to pay to restore data from the archive within a certain period of time. Moreover, if they ended up pulling the wrong archive, they had to start the cycle over. This is a position no one wants to be in: forced to make a tradeoff between speed and budget, especially if you’re trying to recover backups in a disaster or ransomware recovery scenario. 

‍
As Emerick had been a long time reader of Backblaze’s Hard Drive Data and Stats reports, the team decided to see if Backblaze B2 could meet their needs. “I know that Backblaze is a pretty standard name in the industry,” said Emerick, “so that made me more confident in their offerings.” In addition, Backblaze’s affordability and simple pricing was in line with Bethel’s cost reduction goals.

‍

Lessons Learned From Experimentation

Emerick ran a PoC on Backblaze, creating a small set of servers and adding a new archive target on their Rubrik array. He remembers that “it was really a couple clicks, about five minutes worth of work, and we were pointed to Backblaze.” The PoC held up well in terms of performance, throughput, and other criteria. Once the team made the decision to go with Backblaze, all they had to do was point the rest of their servers at the target they’d already created.

‍
“Implementation was pretty drama free,” recalls Emerick. “I was concerned about how hard it would be to switch over from Amazon S3 to Backblaze B2, but it was surprisingly easy and minimal work overall."

A Master Class in Cost Savings and Reliability

Bethel migrated backups to Backblaze over time as files aged out of the Amazon instance and new files flowed into Backblaze. The workflow was all handled by Rubrik, which creates the primary backup of Bethel servers, stores it locally for a short time frame, and then archives it up to cloud storage. The archived backup is then retained for a longer time frame on Backblaze B2 before being purged. The district’s full 125TB+ of data now resides on Backblaze. And, thanks to the cost savings they realized, they were able to reinstate the longer retention period, better protecting them against threats like ransomware.

‍
Bethel turned on Object Lock for the data in their Backblaze B2 Buckets to take advantage of Rubrik’s built-in immutability and give them additional protection against ransomware. “The reason we retain these backups is to make sure we’ve got a good backup that can’t be altered if we do get hit by a cyberattack, so immutability was a key factor,” Emerick explained. 

‍
Also important for the small but mighty team: there’s been little to no maintenance needed. “I haven’t had to do any tweaking or configurations to our Backblaze setup,” said Emerick. “It's really been ‘set it and forget it,’ which is exactly what I need. I've got enough to worry about!”

‍
Storing backup files on Backblaze gives Emerick even more peace of mind. In the very rare instance that he’d need to access a file, he knows that it’s right there at his fingertips. “I’ve found it quick to download a file from Backblaze using Rubrik. In all the tests I’ve run, it’s worked well.”

‍
In addition to reliability, moving to Backblaze has produced savings of 75%, which adds up quickly over the course of the fiscal year. On top of that, Backblaze’s flat pricing structure helps Bethel better control costs and predict spending. “Backblaze gives us a predictable invoice every month, which makes budgeting easier for the school district.” They don’t have to worry about any unexpected charges hitting their books, financially speaking, as more students, teachers, and administrators hit the academic books year after year.