In a recent memorial service for Father Theodore Hesburgh of the University of Notre Dame, former Sen. Alan Simpson had the following advice for attending students: "If you're damned if you do, and damned if you don't... do."
The advice, delivered with Simpson's typical humor, drew laughs from the audience, but it's pretty good advice nonetheless. In IT, we often find ourselves in situations where planning for the future is the choice we would like to make, but making such plans would take away from the work we need to do now. Such is the lot of human beings: We can conceive of the future, but we're pretty bad at actually planning for it.
One example of such planning gaps in the IT world is that old bugaboo, disaster recovery. Everyone in IT knows they should do it, but rarely it gets done. In fact, disaster recovery planning is something that is not getting done in a lot of organizations. But every once in a while, an organization can break the mold and do a better job.
Make an emergency menu
You wouldn't think Waffle House restaurants, a staple of interstate exits throughout the U.S. Southeast, would be a model for disaster recovery, but they are. The 24-hour restaurant chain has drawn up such a good disaster plan for their stores, people started noticing. If there's a natural disaster, such as a flood, hurricane, or tornado, Waffle Houses are pretty good about keeping their doors open and their hash browns flinging.
According to a recent story by NPR, the Federal Emergency Management Agency (FEMA) realized that if a local Waffle House was closed, then things must be really bad.
"In 2011, the current head of FEMA, administrator Craig Fugate, was said to have coined what’s called the Waffle House Index. There are three measures in the index: green, yellow and red... Green means the restaurant is open as usual, yellow means it’s on a limited menu, and red means the restaurant’s closed," NPR reported.
When storms are approaching, Waffle House's corporate teams start watching the weather and renting generators for stores in the area. There's even an emergency menu in place that doesn't include bacon (too much space taken on the grill) or waffles (too much electricity needed).
Although it may seem mundane keeping hot meals going for local residents, it's a good sign you're doing something right with your emergency plan when FEMA takes notice.
You don't have to be FEMA-worthy to have a decent disaster recovery plan, but sitting down and figuring out what you should do when problems arise will save you a lot of trouble down the road.
It is important to understand something right away: Disaster recovery may not mean you will be able to recover your entire business process right away. Waffle House's emergency menu is a good example of that. The first thing to do is establish a prioritization plan that answers this one question: What do you absolutely need to keep your business going? What's your emergency menu going to be?
Paying the tab
Assigning priority is key, because even if you have diligently saved every scrap of data and safely stored it off-site, just like all the disaster recovery specs tell you, you simply may not have enough computing and storage capacity to run that software and data right away. So, what you need to do in your disaster recovery plan is determine the absolute minimum in applications, data, and hardware resources you will need to keep your business running. For example, you will probably need your customer database and all of your front-facing applications, but will you really need HR's database right away? Or your development systems?
After you determine what needs to be started first, keep going on the list, figuring out which systems will be re-activated and in what order. Such prioritization should also be applied to your data. What data needs to be mirrored or backed up in near real time? What needs just hourly or even daily backups? Get your data strategy in order and it will greatly smooth out your recovery process.
Virtualization will also mitigate the resource problem. If your systems are virtualized, they can be brought back online much more quickly and efficiently in a virtual datacenter. Virtual servers also have the advantage of being moved out of harm's way if you have enough time to prepare for a disaster, such as a flood or hurricane.
A disaster is just that: a disaster. And as much as you might want to, you will not be able to turn-key all your systems back on the next day. Bacon and waffles may have to wait.
2 Comments