Relying on Third Parties for DR

Many of us rely upon vendors and third-parties for our disaster recovery efforts. For example, I personally rely upon a refueling company to keep my generator topped off and a maintenance company to keep it running. Other companies rely upon shared data centers, data backup/recovery companies, and DR planners like Agility Recovery.

A weakness in these plans occurs when a regional disaster impacts multiple companies. In these scenarios, the third-party may lack the capacity to handle all the requests and be overwhelmed. One thing that happened to my data center during the winter storm (which knocked power out for five days) was that the fuel trucks were delayed for 48 hours, and the maintenance crew delayed for 24. The times are well within tolerances, but well beyond normal service levels.

Many Michigan data centers are on generator power this morning. From the Associated Press, “Wind gusting more than 60 mph knocked out power to about 413,000 Michigan homes and businesses on Sunday as temperatures dipped back into the 20s and 30s.” AT&T Wireless was out state wide yesterday. Level 3 Communications, a backbone Internet provider, also had an outage yesterday morning in the Detroit NOC.

How are your disaster recovery strategies faring?

Business continuity and disaster recovery have been on my mind a lot lately. The SNW conference is fast approaching and I am putting the final touches on my slide deck. One question is when and where a company should open a backup data center.

First, and I cannot stress this enough, do an impact analysis. Do you really need another data center? The text book example is the company, in an earthquake zone, which determines that bolting server racks down and buying additional insurance provides the same level of protection at significantly less cost. Your organization does not operate in a textbook, of course, and you may very well need another data center.

Having made the business case and established the budget, the next question is where to locate the facility. The following should be researched and considered:

• Access – road, rail, and air, telecommunications
• Proximity to current data center (under 30 miles makes real-time fail-over possible)
• Local crime rates (history of protests, strikes, or riots)
• Municipal services (police, fire, ambulance, power)
• Wind patterns (is this downwind from nuclear power plants or military targets?
• Weather patterns (hurricanes, tornadoes, et cetera)
• Geophysical conditions (fault lines and earth quakes)

Gather all of this information and begin looking possible sites. Look for sites that are within budget and near high speed Internet backbone links. Narrow these down to those with redundant power distribution points. Then consider such things as wind and weather. This should narrow the possible sites down quite a bit. Then begin considering how your organization will transport people to this location. Airlines are best, but flights may be grounded in a widespread disaster, so also look for wide accessible highways.

In the end, come up with the punch list of three to five ideal sites. Go out for a site visit and confirm your assumptions. Some sites might not make the cut upon visiting, as your information may be out of date. At the end of this process, you will have done the homework and due diligence to make the recommendation to senior management.

What is your budget for disaster recovery?

Are you spending too much?

Or too little?

Ideally, Disaster Recovery is a program that contains one or more strategies. Each strategy is a specific way to recover IT systems for one or more business processes. For example, you have a hot site strategy with one-to-one duplication of the hardware and software used in production. This strategy is costly and so it protects the critical business processes. For non-critical processes, you may have a cold site strategy. Basically, you’ll buy new hardware and restore from tape should an outage occur.

There is a means to calculate the budget for Disaster Recovery.

Step one is to determine the likelihood. Map the IT software and hardware to the business process. Determine the threats you are protecting against (fire, flood, earthquake). Do some digging to estimate the how likely these threats are to occur (Annualized Rate of Occurrence).

Step two is to determine the financial impact. If the impact occurs, everything is offline, the business process grounds to a halt, how hard will that hit the business? Quantify the impact in terms of dollars (Single Loss Expectancy).

Step three is to multiply the two to determine the Annualized Loss Expectancy. ALE = SLE * ARO

ALE is represents the business’s asset exposure. It is the most that should be spent on a Disaster Recovery strategy that mitigates the risk. If ALE is $50,000 and the recovery strategy costs $100,000 a year, then obviously you are spending too much. If you are spending $10,000 a year, you are either a hero or putting the business process at risk by using an insufficient strategy.

I find that organizations who are just starting out with Business Continuity and Disaster Recovery greatly benefit  from this budgeting method. It demonstrates a clear link between assets and protection. This way, an IT team can cost justify investments in Disaster Recovery systems.