Pittsburg State University
From Cyclones to Cybercriminals: How One University Protects Data in Tornado Alley
Pittsburg State University
Tim Pearson
Director for IT Infrastructure and Security, Pittsburg State University
Data Protection
Mbps Via I2PX
Day Set Up
Pittsburg State University deploys a private cloud with two nodes on-premises and a third across the state for geographic separation. The system affordably protects them against the tornado threat in Kansas, but the techniques that work for disaster recovery don’t necessarily work for data integrity. They wanted to add another layer of protection against ransomware and other threats with air-gapped backups.
The university added Backblaze B2 Cloud Storage to their already robust system using Object Lock to create a virtual air gap. They send nightly backups to Backblaze B2 via the Internet2 Peer Exchange (I2PX) service, protected by Object Lock so that no one can modify, delete, or encrypt the data–giving them peace of mind at a time when educational institutions are frequent ransomware targets.
Backblaze B2 effectively and affordably fit into the university’s private cloud infrastructure, allowing them to achieve 4x protection by adding off-site backups. The Internet2 network’s settlement-free I2PX service—peered directly with the Backblaze B2 Storage Cloud—enables them to protect their data faster, with less latency, and at a lower network cost. And their budget isn’t tied up with a diversified cloud provider or associated transit costs, so they can respond to changing demands flexibly.
Pittsburg State (Pitt State) University was founded in 1903 as the Auxiliary Manual Training Normal School, originally a branch of the State Normal School of Emporia. Over its long history, it served as a teacher’s college before expanding to a fully-fledged university in 1977. Today, it serves more than 6,000 students plus faculty, and offers some technically demanding programs including construction, nursing, and polymer chemistry. The university is a member of the Kansas Research and Educational Network (KanREN), the regional network serving the state’s research, education, and other community anchor institutions and providing access to Internet2.
As the director of IT infrastructure and security for Pittsburg State University in Kansas, Tim Pearson knows all too well the absolute necessity for universities to invest in comprehensive data protection. However, unlike most IT directors, he has some added insight that comes from the other hat he wears: that of a fire chief.
Pearson has volunteered as the fire chief for Carterville, Missouri for several years. He was on the ground the night an intensely destructive tornado struck Joplin, Missouri in May 2011. “We were up close and personal with the destruction, and being an IT guy, I was also in contact with a lot of my peers in Joplin who had lost their data centers,” Pearson recalled.
While lots of areas of the country face the destructive potential of strong tornadoes, and most areas are at risk of at least some kind of natural disaster, Tornado Alley is uniquely at risk. States in the heart of the region rank in the top 10 list of states with the highest average number of tornadoes per year. That’s where Pitt State sits, right in the southeast corner of Kansas, second on the list. Disaster planning is non-negotiable, and Pearson knows that better than anyone.
As a self-professed “belt and suspenders kind of guy” when it comes to data protection, Pearson takes a no-excuses approach based on years of experience and near misses. “I have been in IT long enough to remember when hard drives were nowhere near as reliable as they are today. When I started my career, we had some real near-disaster scenarios, and I vowed I would do everything I could to prevent that from ever happening again,” he said.
...we had some real near-disaster scenarios, and I vowed I would do everything I could to prevent that from ever happening again.
Tim Pearson, Director for IT Infrastructure and Security, Pittsburg State University
For Pearson, preventing disasters started with deploying private cloud infrastructure via Commvault Distributed Storage that could withstand the threat of a tornado while also aggressively managing costs.
He established a three-node system with two data centers on campus and one across the state at another university within the Kansas Research and Educational Network (KanREN)—a regional network that provides access to Internet2, a non-profit organization that delivers an incredibly high-speed, low-latency national network for research and educational institutions. The two on-campus nodes run an active-active split with half of their compute infrastructure running on each node, but both data centers have enough horsepower to run their entire workload in the event one were to go down. The third node hosts a third copy of their data for geographic separation, and enough compute horsepower to run critical services should they need to.
The three nodes sync in real time, so Pitt State has three updated copies of their data as production runs throughout the day. “We didn’t want to depend on the previous night’s backup should anything terrible happen,” Pearson noted. They have real-time coverage throughout the day right up to the moment of failure, and they’d have to lose three different locations, one of which is across the state, to incur any data loss.
Notably, they implemented it all at a fraction of the cost of using a diversified public cloud provider for all of their needs. “We think there needs to be a business case for moving to the cloud,” Pearson explained. “Which is why we deployed a private cloud instead of moving our workload to Amazon. We’re saving over six figures every year by deploying our own private cloud with a similar level of survivability.” Pearson readily admits he has a sign on his office wall that reads, “There is no cloud. It’s just someone else’s computer.”
With his innovative private cloud in place, Pearson felt disaster recovery was well in hand. Even if a mile-wide tornado took out both on-campus data centers, the third node would allow them to continue functioning. The next challenge Pearson sought to solve was data integrity: What happens if someone accidentally deletes half a million records or the university suffers some kind of data breach?
“The techniques that work well for disaster recovery don’t necessarily work well for protecting the integrity of data,” Pearson explained. He had good reason to be concerned. Universities store highly sensitive, personally identifiable data, and cybercriminals know it. Emsisoft, a cybersecurity company, reported that over 2,000 educational, government, and medical institutions were hit by ransomware attacks in 2021. That figure includes 26 colleges. The attacks also caused operational interruptions and at least 118 data breaches.
For the task of protecting data integrity, “Nothing can beat immutability at the file system,” Pearson said. In short, he wanted to create immutability with an air gap between backups and the active network by storing data on something that was either physically or virtually isolated.
Pearson explored network-attached storage (NAS) solutions which he would then have to physically disconnect from the network to create a physical air gap. It appealed to his do-it-yourself mentality but he knew the manual lift would be cumbersome. Though he avoided cloud-based solutions in the past, he recognized this as an area where the business case was strong. He started looking into cloud storage to strengthen Pitt State’s data integrity.
The worst scenario I can think of is if all of our other protections fail…and a bad guy somehow gets my backup admin’s credentials. Even if that happens, with Object Lock, they can’t alter the data.
Tim Pearson, Director for IT Infrastructure and Security, Pittsburg State University
Pearson didn’t have to look far to find a cloud solution he thought might work. As an amateur videographer, Pearson already used Backblaze to back up his personal files. He knew it to be easy and cost-effective. He reached out to learn more about storing Pitt State’s backup data in Backblaze B2 with Object Lock.
Object Lock is a powerful backup protection tool that prevents a file from being altered or deleted until a given date. It stores data using a Write Once, Read Many (WORM) model, meaning after it’s written, data cannot be modified or deleted for a defined period of time set by the admin at the outset. Any attempts to manipulate, copy, encrypt, change, or delete the file will fail during that time. The files may be accessed, but no one can change them, including the file owner or whoever set the Object Lock.
That level of protection appealed to Pearson. “We always try to look at worst case scenarios,” he said. “The worst scenario I can think of is if all of our other protections fail—multifactor authentication, login groups, firewalls, VPNs—and a bad guy somehow gets my backup admin’s credentials. Even if that happens, with Object Lock, they can’t alter the data.” Convinced it provided the data integrity protection he was looking for, Pearson set up Backblaze and integrated it as a destination for his Commvault backups in less than a day. Now, Commvault de-duplicates the data, then sends a copy to Backblaze nightly where it’s stored in a bucket protected by Object Lock, giving them four times the protection combined with the three nodes on-premises. And, by sending their data through KanREN to the Internet2 Peer Exchange with Backblaze B2, Pitt State is able to protect their data with less time and effort.
We’re trying to stay agile enough that we’ll be able to take advantage of opportunities as they present themselves, and adding Backblaze was one of those opportunities.
Tim Pearson, Director for IT Infrastructure and Security, Pittsburg State University
For Pearson to invest the university’s dollars in a solution, there had better be a bulletproof business case, especially when it comes to moving anything to the cloud. For this specific use case, Backblaze B2 Cloud Storage stood up to Pearson’s measured decision-making process. It effectively and affordably filled the university’s need for two key pieces of the data integrity puzzle—immutability and geographic diversity. “Backblaze B2 had the capability we lacked,” he confirmed. “I bolted it onto our system, so now I have off-site backup that is safe and well-protected from even a regional disaster in Kansas as well as ransomware protection with Object Lock.”
Now, Pearson is free to focus on the next puzzle—expanding the university’s compute footprint to harden up their disaster recovery even more. What’s more, Pearson says he appreciated Backblaze’s responsiveness and overall culture. He cited the Backblaze solution engineers’ enthusiasm, outside-the-box thinking, and plain-spoken approach as traits that made the process a pleasure instead of just another box to be checked off, or worse, an exercise in corporate puffery.
In addition to the obvious and direct benefits of redundant backups and data integrity, Pearson says Backblaze along with the Commvault private cloud is insurance against future shifts in the wind (not to mention actual tornadoes). “There’s a lot of upheaval as higher education tries to figure out what’s going to be the new normal for delivering quality education to students,” he said. “We’re trying to stay agile enough that we’ll be able to take advantage of opportunities as they present themselves, and adding Backblaze was one of those opportunities.”
Commvault provides enterprise data protection and data management software solutions. Their product lines help businesses with backup, archiving, and organizing data.
Internet2® is a non-profit, member-driven advanced technology community founded by the nation’s leading higher education institutions in 1996. Internet2 serves 335 U.S. universities, 58 government agencies, 46 regional and state education networks and through them supports more than 80,000 community anchor institutions, over 1,000 InCommon participants, 59 leading corporations working with our community, and 70 national research and education network operators that represent more than 100 countries. Internet2 delivers a diverse portfolio of technology solutions that leverages, integrates, and amplifies the strengths of its members and helps support their educational, research, and community service missions. Internet2’s core infrastructure components include the nation’s largest and fastest research and education network that was built to deliver advanced, customized services that are accessed and secured by the community-developed trust and identity framework. For more information, visit https://internet2.edu.
