June 20, 2007 — CIO — Is your business ready for a visit from Erin, Felix, Jerry or Karen this summer? Those are the names of four of this year’s storms that will be aiming to hit shore this season. The National Oceanic & Atmospheric Administration (NOAA) is predicting three to five major Katrina-sized, Category Three hurricanes, with up to 130-mile-per-hour winds and 12-foot storm surges, in the coming months.

According to market research firm In-Stat, most companies are not prepared for disaster, whether it comes in the form of gale force winds and rain, power outages, a health-care threat or terrorist attack. A December 2006 report from In-Stat reveals that only 28 percent of enterprises had fully implemented disaster recovery applications, and that 20 percent either had no continuity plans at all or were unaware of any plans.

1. Identify vulnerabilities in your business. Imagine what could happen if a disaster shut down your company, and assess where your vulnerable spots are if a hurricane strikes, a pandemic hits or something else shakes your organization’s foundation. To determine how your business might be impacted, gather a group of individuals from the business and IT who intimately understand business and technical operations to do some scenario planning, advises Bill Nagel, an analyst at Forrester Research. He also recommends bringing in an expert in business continuity planning to help identify vulnerabilities you and your team might miss. Scenario planning will help you identify additional infrastructure you may need, such as backup power generators, redundant data circuits, backup applications and remote data storage.

2. Replicate your data and ensure redundant systems. Your continuity planning should include system redundancy to support vital business applications. Sub shop chain Quiznos needs to maintain system uptime and guarantee that little to no data is lost during severe weather, according to Michael Derosier, the Denver-based company’s vice president of IT. Denver is free from hurricanes and earthquakes, but other weather sometimes causes power trouble for Derosier’s group. “We plan for everything from blizzards to pandemics to other major catastrophes,” he says.

To ensure the rapidly growing outfit doesn’t skip a beat, production applications get replicated every night in the company’s disaster recovery center, which is about 20 miles from headquarters. Replicating makes restarting after power outages or other network troubles a snap. It also ensures that no data is lost and that the most up-to-date versions of data and applications are stored.

Earthquakes are a particular concern for Steve Davidek, operations and systems administrator for the city of Sparks, Nev. “We think citizens and families first,” says Davidek. “So our biggest continuity issue is making certain our police and fire dispatch systems are protected.” To that end, the city virtualized its data systems by putting them on a storage area network (SAN) this past January. The city can switch over to the SAN automatically if its emergency systems go down. With the police and fire dispatch systems on the SAN, dispatchers will never lose touch with vital name, address and telephone number data that police and fire departments need for responding to calls.

3. Implement a multi-modal communication system. When the BlackBerry messaging system went dark for 12 hours this past April, many companies were left figuratively bumping into furniture and groping for a light switch. A business functioning primarily on one communication mechanism (such as the BlackBerry) should consider implementing a multi-modal communication system for redundancy and continuity’s sake, as Fairleigh Dickinson University in New Jersey did.

Fairleigh Dickinson’s multi-modal communication system enables the university to notify its 12,000 students and staff by e-mail, text message, voice messages or a combination of those methods about snow days, flooding, major schedule changes and emergency campus information. “We realized that not all of our system users check e-mail constantly or consistently, so we needed a system that could broadcast to e-mail, cell phones or home phones,” says Neal Sturm, CIO at Fairleigh Dickinson. “We had to plan for ways to be absolutely certain to get messages out if a disaster struck.”

A strong communications backbone is critical when dealing with unexpected events. Your continuity plan should include a private IP network for communications, according to Forrester Research. If this seems like overkill to you, your CEO or CFO, consider what might happen in a health pandemic during which your company’s regional or national workforce is forced to stay home: To remain productive, they need to be able to access business systems on the corporate network from home, which would require your company to have a private IP network, VPN connectivity or some other type of dedicated Internet service. With such technology in place, employees can continue to work, and executives can continue to engage in high-level daily discussions and decision-making about business operations.

4. Document procedures, train employees and test the plan. An untested plan is only a half-step above no plan at all. An April 2007 report from Forrester Research states why IT leaders must be strong advocates of regular continuity and disaster plan testing: “Aside from an actual disaster, it’s your only chance to ensure that everything works the way it should. The middle of a disaster is not the time to discover your plan isn’t up to scratch.”

Quiznos’s Derosier says his company tests its continuity plan and disaster recovery system quarterly. Specifically, Quiznos checks the viability of its off-site applications by running test scenarios on the applications for each functional business unit. The company will run a dummy royalty franchise payment through the backup applications system, for example, or add a fictitious diagnostic employee to the testing roster to evaluate how the backup financial and accounting systems work. The city of Sparks, Nev., has implemented regular testing as well.

Sparks’ Davidek says that documenting business continuity procedures in the plan along with employees’ tacit knowledge of those procedures should go hand in hand with testing it. There’s nothing worse than discovering that the only person who knows how to reinstall vital applications is on vacation. Something similar happened to Davidek a few years ago while he was on vacation. He had to quickly talk a systems administrator through the time-sensitive process of re-installing RAID drives (multiple disk drives that are combined into a single entity) over the phone because two had simultaneously failed and the systems administrator on site didn’t know how to fix it. Had that knowledge of installing RAID drives been documented and shared, the situation would have been solved more quickly and with less stress. Now Davidek ensures that his staff shares their knowledge. “We cross-train people in our department to spread out among people all of those little things you learn from experience,” he says. Other disaster experts advise building online documentation and knowledge center repositories that can be accessed quickly and remotely if necessary.

5. Update the plan. If you don’t update your plan to reflect changes in business operations and business processes, your business continuity plan will not work when you’re forced to use it because it will not reflect the way the business currently runs. For example, the plan should be revised with every new product or service offering, according to Quiznos’s Derosier. Without updates, new offerings may not be covered if the plan is applied, he says. Similarly, discontinued services should be removed from the plan so that it doesn’t attempt to protect something that is no longer there. Some companies, like Quiznos, hire a full-time business continuity specialist to update the plan once it’s created. 

If thinking about the biggest “what if” questions seems melodramatic, consider the statement Bill Proenza, director of the National Hurricane Center, left on NOAA’s Hurricane Center website: “Preparation through education is less costly than learning through tragedy.”