Back from Disaster

—

Just a few days before the start of a recent school year, Beaverton School District in Oregon experienced what officials there now simply call “The Event.” 

When IT staffers arrived at work on the Friday before classes were scheduled to begin, every server in the district’s data center was offline, and they couldn’t identify why. When they tried to reboot, about half of the hard drives failed, including those dedicated to the district’s student information system, email, finance system and a number of other applications. At first, staffers thought the issue might be related to the data center’s uninterruptible power supply, but they couldn’t find anything wrong. With the district staring down the prospect of starting the school year without a number of its critical applications, IT staffers canceled their end-of-summer vacations and dug into the problem. 

Eventually, officials discovered that the fire suppression monitoring system protecting the district’s IT infrastructure had malfunctioned. A nozzle in the system, which was recalled years ago, caused vibrations in the room. For hard drives spinning at 15,000 RPMs, it was like a sonic boom. 

Gas canisters discharged throughout the data center to stop the fire, even though there wasn’t a fire. An alert was sent to the school’s alarm company, but the fire department wasn’t notified; the district’s office of public safety had instructed the company to call the fire department only if two alarms went off as a cost-savings measure. Any system of protection that could be compromised, had been compromised.

IT staffers worked around the clock to bring systems back up on the remaining hard drives, restoring applications from tape. By Sunday evening, just two days before the start of school, they finally had the student information system and most other critical applications up and running. 

“We thought that was the end of the crisis, and that we’d pretty much mitigated the problem,” recalls CIO Steven Langford. “Everyone was exhausted, and I told everybody, ‘Go home and sleep.’ What we didn’t know was that the problem hadn’t even started yet.” 

—

A Shifting Mindset

Shortly after the worst of the crisis had passed, Langford met with CDW field representative Angela Gadient, who is now a field sales manager for K-12 West. “We started looking at the systems that we had around protecting our data and data center, and we realized we needed a lot of help,” Langford says. “That’s not our expertise.” 

Gadient arranged for CDW experts to hold a one-week workshop for Beaverton officials and employees. As the CDW team listened to stakeholders describe their existing operations and their concerns about data security, it became clear that the district needed to move away from thinking solely in terms of IT solutions and toward the problem of disaster recovery. “Instead of saying, ‘Hey, why don’t we help you with some servers and storage,’ it was clear that it was a much bigger systemic issue,” says Gadient. “We started having conversations around risk, and how organizations can address this kind of risk. It isn’t just a matter of servers and storage. It’s a matter of policies and procedures, as well.” 

Langford says that district officials were impressed with CDW’s broad view of the situation. While other vendors and resellers tended to talk in terms of only the data center, he says, CDW’s experts focused on ensuring continuity of operations for the entire school district. “It really exposed us to the business continuity planning piece of things, which is larger than disaster recovery. And it showed us where the data center planning would fit into a holistic organizationwide plan.”

Gary Arnce, who now works as a regional manager of data center solutions in the Western U.S. for CDW, was a data center architect at the time, and worked closely with Beaverton officials as the district worked through a months-long engagement aimed at identifying, prioritizing and protecting its critical applications. 

“Very few organizations, especially school districts, know where their data lives, or have service-level agreements (SLAs) outlined for every application,” says Arnce. “It’s a different mindset from how most K–12 districts are used to approaching disaster recovery.”  

In part because of how spectacularly their existing processes failed, Langford says, school district officials were receptive to a new way of thinking. “We had been investing in our data center, in terms of backup technologies and database snapshot tools,” he says. “So, we had those systems. Our challenge was that everything was in the same room, except for the tapes. So, it really made us start thinking about what a better approach would be.” 

—

Redundancy and Responsibilities

Beaverton’s disaster recovery and continuity of operations engagement with CDW resulted in a plan that ensured multiple levels of backup outside of the district’s primary data center. As part of a school construction bond, Beaverton built a secondary data center in an existing district facility around 1.5 miles from the primary site. The new facility ensures that the district can keep critical applications online if its primary data center faces an isolated outage (such as the malfunctioning fire suppression system that crippled the district before). However, it is too close to the primary site to protect against a larger disaster, such as a major earthquake. Partly because of this proximity, Beaverton schools are also working to implement a cloud disaster recovery solution. 

“We have redundancy at the data center level locally, and then we’re looking at the cloud as our disaster recovery and business continuity,” Langford says. 

Perhaps just as importantly, the district has codified its practices for backing up data and maintaining those backups, and has written service-level agreements outlining which applications will come back online in certain time frames after a disaster. The engagement with CDW•G also resulted in a runbook that provides guidance for a number of different scenarios (such as how to issue paychecks if a disaster prevents human resources employees from coming to work). The runbook incorporates details down to the level of vendor phone numbers, ensuring that district officials have all of the information they need to navigate a crisis. 

Gadient notes that the school district’s initial crisis was triggered by an employee mistake, and says that the district’s new plans are designed to mitigate the impact of human error. “Beaverton thought they had disaster recovery in place already,” she says. “But what they learned is, because they didn’t follow best practices, they were harmed. Now, they’ve vetted their responsibilities and priorities. The next piece is practicing the plan with tabletop exercises. Those are the things that are going to be critical to their success, and those are the things that school districts often don’t think about.” 

“If there is a data center failure today, everybody in the district knows what needs to happen,” Arnce says. “Before, they were scrambling. Now, they know those tier-one applications are going to come up in that secondary data center, and they’ll be up and running.” 

Langford says the district is updating its disaster recovery and continuity of operations plan annually, and that staffers now feel prepared to execute it if necessary. 

“Hopefully, we’re never going to need it,” Langford says. “But the planning work was very valuable in increasing staff confidence that we’re going to be ready should something happen. If this happens again, I know I’m going to have my systems up, I know how fast we can do it, and I know what data we have that we need to recover.”