Nearly 50,000 swimmers nationwide rely on the U.S. Masters Swimming organization for training, competitions, and various local programs. Started in 1970, the organization has grown to support 500 local clubs and its website, www.usms.org, is the hub that helps manage this large, distributed nonprofit.
Yesterday, during a routine import, the database containing all the member records became corrupted. Access to the website was shut down to prevent additional errors while the team tried to restore the systems, but to no avail.
Luckily, the USMS team understood that data gets lost and rigorously prepared for such an event. Data was backed up in three ways: incremental, transactional, and tape. Even so, it turned out their first two backups failed. Fortunately, the third backup worked, and the organization lost three days, not 30 years, of data. The USMS team says:
“For the next six hours, five experts attempted to recover the corrupted data and it turned out to be impossible.”
Continuing to show diligence and savviness, the USMS team has developed their plan to go forward:
What we are doing short term
We are going through all auxiliary sources of data including credit card transactions, automated emails, and batch processes. We have hired extra help and we are manually copying the data from these sources as soon as we receive it.What we are doing long term
We are purchasing additional hardware and leasing a separate facility to do real-time secure backups of transactional data thereby effectively adding a fourth set of backup data that is accurate up to the last transaction. In addition we have established a new set of protocols to further mitigate potential data loss.
If there are some lessons to be learned from the USMS experience:
- Make sure your data is backed up.
- Back up to more than one place (e.g. local hard drive and online backup service).
- Periodically check your backups.
You can never be too careful with your precious data.
