I would suggest, instead of trying to write a DR plan for the big one, you tackle something much smaller and more valuable in the long run.

A critical requirement for any contingency plans is backing up the data.  Go find out if all the production systems are backed up and the back copy is taken offsite.  If you’re lucky to be doing a disk to disk backup and the duplicate is going over the wire to another site, cheer loud and long.  The biggest hurdle has been gotten over.  If not, there’s a place to start that will probably be fairly easy to get implemented.  Work with your IT staff to get adequate data backups taken off site.  With those backups, your heroic and over worked IT staff can get your production systems recovered from any size of disaster and those backups are a direct benefit to the company, since even the Director of IT will agree that taking backups is a good idea.

Another smaller task to handle is writing down current incident handling procedures, standardize how they’re handled and what is needed.  Disaster’s come in many flavors and size but the response is standard.  Either it’s something that can be handled in house with your current resources or you can’t.  Since there will be an in house disaster [server crash, data corruption, bad patch, network going down] make sure the response is consistent and the service level agreements are in place and financial support covered.

Start small, with those production requirements that make the biggest impact when something goes wrong, to set the standard.  Take a look at your notification system.  See if you can’t get a 3^rd party notification in place and in use for incident notifications.  Auto notification systems can be used for lots of purposes besides calling for help in a disaster.  You can send mass notifications to employees for PR purposes, let them know about important company messages, even contact customers and vendors.  A good automated notification system has lots more purposes that just sending out disaster messages.

Another item to follow up on is single points of failure.  Remediating them might cost money but they’re the kind of fix that’s useful all the way up to the big disaster, so you’re more likely to get them tackled that getting another data center built or funding a 3^rd party offsite recovery location.  Duplicate networks, external phone lines, backup generator and UPS system, production server clusters and more exist that are far more likely to be needed than a DR plan for an offsite recovery.

In my experience, most senior execs either don’t know what’s involved or don’t care.  For those who think they know what’s involved and who are only mandating contingency because of some government directive, you’ll never get support, no matter how good your argument might be.  If you’re really lucky, and high enough in the food chain to get a chance to present your case to the BOD, keep in mind that education of what’s involved is key to getting some support and guidance.

In the meantime, since you’ll probably never get a chance to speak your piece to the person or persons who will really make the necessary decisions, I suggest you tackle this a different way.

Let me back up a bit and offer some considerations for you to think about.

What is really the purpose of your DR or BC plan?  In the old days, back when the industry got started, DRP’s were about mainframe computer recovery plans.  The way a DR plan was written required another mainframe be sitting in a distant computer room somewhere, just waiting to be brought up.  The description of cold site, warm site and hot came about from how quickly a mainframe computer could be installed, configured and booted.  The time frame for recovery was contingent upon whether or not the equipment existed in that distant data center.  Losing the data center was a financial blow but it was still possible to manually run the business in the short term.  The business was likely to be able to continue operating during the 1 to 7 days it took to recover.

Now fast forward to our current business operations.  Imagine running your business without computers.  Could you do it?  Maybe.  If you’re a small service industry, operating locally out of an office or store front with just a few employees, you can probably manage without the computers.  Now grow a bit.  Throw in a bit of manufacturing and a couple of servers.  Still think you can get by without computers for a week?  For most businesses, the answer is NO.  Not because it wasn’t done in the past but because you’ve become dependent upon computers and you don’t have the training or manual equipment to replace using computers.

Now, 99% of contingency planners talk about being ready for the big event, that disaster that’s going to wipe you out.  They’re right, if you’re a small to moderate sized business, the big one is highly likely to wipe you out.  Why?  Because you couldn’t recover you’re ability to handle the needs of your business and customers in a timely fashion and the unexpected financial loss wasn’t covered by insurance.  Large corporations have the manpower and financial resources to recover but how well they recover is indirectly dependent upon how well you do your job.  Large corporations have the same unsung hero staff members that smaller companies do.  The advantage is they have more of them and they can hang in there longer to allow the heroes to work themselves into an early grave getting the corporation back on its feet.

Small to moderate businesses don’t have that luxury of staff and finances, so planning for them has to be more practical and definite.  They can’t afford another data center or the cost of a third party recovery site contract, so they have to chose a different recovery path, one that will provide greater support with less cost.  Small to moderate businesses also have the great advantage of having senior executive support and direct interaction and decision making.  They can and should have contingency plans without all the politicking that larger corporations have to go through to get a simple decision made.

Well, as you talk to people, you’ll collect more lists.  I track all the ‘lists’ I’m given in a spreadsheet with the information on who gave them to me, what they are used for and how they’re maintained.  Do I print them?  Include them in a plan?  No and no.  They’re supporting information.  Helpful when I talk to managers about what needs to be recovered but useless otherwise.  (I can hear the outrage from here but it’s true.)

Sub rant:  Any list manually maintained will have something out of date.  If you really need a list of what’s in the data center, lobby hard for some software tool that will auto discover the information and generate reports.  Why?  It will be far more accurate, current and usable than a manual list and you can run the report yourself.  So if you really want some lists to wave around, at least they will be reasonably valid.  Also, the IT staff will like the tool and keep it running, since they’ll use those reports as well.  Killing two birds with one stone here folks, three if you count the SOX compliance folks

What other kind of information winds up in the traditional plan?  What the plan is supposed to cover.  Otherwise known as ‘Purpose, Scope, Assumptions, Gaps, Risks’.  Yikes, how is any of that valuable for recovery purposes?  None.  What is it used for?  Reports.  The kind you give to auditors and managers to make them happy.

Yep, reports.  Those obsequious, time consuming (time wasting really but you can’t tell a manager he’s wasting your time), useless reports just to prove the company is doing something.  I wonder, when did writing reports replace doing real work?  Of what value does a report that details what the company isn’t doing provide for a ‘recovery’ plan?  None of course, which is why I call it a supporting document and recommend you keep a folder just for reports.  Auditors will love them, managers will be happy and you can copy and paste when it comes time to make the next version.

Of course all the information in that report was gotten verbally, which is why you need to take notes when you ask questions, so you can provide that report quickly, impressing your boss and keeping your job.

The other ‘report’ that shows up in a traditional plan is really the company BC charter, continuity policy and DR procedures.  Yes, these types have value, if only because they satisfy the Federal or State or insurance or other bureaucracy requirement to have them.  If the company doesn’t follow and enforce them, they aren’t worth the paper they were written on.  Don’t know what should be in them even after you read that BCP book?  Download a selection from the Internet and fill in the blanks.  If your company is serious about them, the committee that reviews them will revise them appropriately.  If you’ve given them a reasonable starting point, you’ve done your job.  If the committee that accepts them doesn’t question them, well the auditor or exec with a grudge will have ammunition for their purposes and you’re good in somebody’s book, which should count for something.  If not, you’ve still done your job.  That’s what they’re paying you for, right?

Yea, I’m familiar with them too.  You know what the problem is?  Book learned plans fails to distinguish between general information, reports and recovery instructions.  I call it the difference between supporting documentation and recovery documents.  (I’m developing a fondness for this blogging gig.  I get to write down all my favorite grips and solutions.  Cool!)

The traditional, certification level standard for a business continuity plan includes huge amounts of data that is subject to rapid change.  The ‘standards’ fail to distinguish between information used to develop a recovery strategy and actual recovery instructions.  There’s a big difference between the two types of information.  Your ‘recovery’ plan should only contain recovery instructions.

Sounds simple, no?  Actually, once you learn the difference between types of information and the purpose it’s used for, it is simple.  It’s also a lot less work for everyone involved in developing a recovery plan.  So let’s start with the information gathering phase of business continuity.  I’m going to use a data center recovery plan for this discussion but it applies to other facility type recovery plans.

First, someone decided the business needs to have a recovery plan for the data center.  Great.  What do you need?  Well, you need to know what the data center does.  The useless answer is: data processing.  So of course some wise and all knowing person says: computers and applications.  Better, a little.  So you go pester the data center manager for a list.  The kind and helpful data center manager says sure, I have a list of computers and applications.  It’s a little out of date but here it is.  Really it’s a lot out of date but nobody admits that until they’re sure that’s what you want to hear.  You want a hard copy printout?  The traditionalist type planner says sure, that’d be good, thanks.  The practical planner says “send me the link”.  We’re talking email here and a spreadsheet of information that stored on somebody’s personal computer, or if they’ve had this problem before, some common electronic storage device in the data center.

So you now have your first piece of information.  Is it usable?  Not for offsite recovery.  Do you need it?  Sort of.  This is the first piece of supporting data collected and it often ends up in the traditional plan as an appendix.  It’s out of date before it was even printed (you knew that right, but you printed it anyway) and even more out of date when the ‘plan’ finally gets officially published.  One of the things that most planners fail to consider, is that data centers are so specialized that a simple list of equipment is useless for anybody but an auditor or accountant.  You want to recover your ability to run the applications that your employees and customers need.  For that, you need a whole different approach.  Since this is a rant about documentation, not recovery strategy, I’ll table that discussion and get back to documentation.

What constitutes a real disaster?

Disasters are by definition, something devastating that happens to the other guy’s data center, not yours!  Right?  Oops!  You have backups of your data, right?  You’re covered, right?  Ah, did anyone happen to back up that development system you’re using for that new software that’s only cost a million so far?  After all, when that over-write was done, and all that time consuming code was wiped out, they took a backup before they started, so you’re good.  Oh, they don’t back up the test and development systems because they’re not production systems?  Oh well, you can spend another million and a year to re-write that code.  At least you don’t have to start from scratch.  Was that a disaster?

Well, not technically.  Nobody dies, the data center wasn’t impacted, production stayed up.  Of course, the manager who has to go back to the board and explain why they’re six months behind and starting over may consider it a disaster, particularly if it’s his job on the line when the next layoffs roll around because of that little oops.  After all, the PR department has been promising customers the new applications, the investors were expecting increased income and the board was planning on how to spend that additional income that now isn’t going to happen.  So maybe a little time spent putting in place some backup jobs to ensure that even non-critical systems can be recovered would have saved a lot of money and headaches, not to mention heartache on the part of the developers/programmers who had to re-write all that code.  And the company reputation, that poor IT manager’s career, the financial loss and the trust investors would have in the business the next time they announce they’ve got another application in the works.

I’m exaggerating, surely?  Not hardly.  All those things have happened in businesses I’ve worked at or knew personally some IT person where it happened.  I was one of the people who got the call when the power outage shutdown only the data center.  All that careful planning to ensure the data center had power when the city power went out introduced a weakness into the system.  Some ground fault, never actually located, just bypassed, shut down only the data center.  Fortunately, we had a disaster recovery plan (it was intended for the DR site) that could be used to re-start the normal production system when the power came back up.  Even so, it took two days of round the clock heroic effort on the part of the IT staff to get production up.  They were still finding things wrong in the test, development and staging systems for the rest of the week.  They got to plan the next power outage the following weekend when the electricians came back to install a temporary emergency fix to (hopefully!) prevent it happening again.  Anybody heard of a single point of failure analysis?  Helps if you pay attention to what’s found.

Is your datacenter insured?

The broken water main that floods the data center and closes the building for two and a half days while they get the water pumped out from 90,000 square feet of raised floor space.

Why did the architect run a 4” water main over the data center in the first place? I’ve seen it in 2 different buildings on opposite coasts.  And why didn’t someone blow the whistle before they built the building? The power outage that only shuts down the data center (that has happened too!) and nobody knew in what order to start all the inter-related applications, they’d never done it before. My favorite, never gonna happen event; the power surge that sets a mainframe computer on fire because the part that would prevent it was faulty and removed so the mainframe computer could continue to operate until the part came in. It really happened, too.  The part was removed Thursday evening after batch was run, power surge came through Friday morning, 8:35 AM (some driver hit a transformer pole, 50,000 volts went through the lines before another transformer blew), the mainframe computer catches on fire, taking the whole data center down while lots of equipment is replaced over the weekend.  By the by, the replacement part arrived Monday morning on schedule but by then IBM had replaced the entire mainframe computer. (It only cost $125,000,000.00 originally, chump change, right?)  Somebody paid the bill for that one-in-a-million event.  Hey, it wasn’t a total disaster, just a hazard of assuming a power surge would never happen while the surge protector was removed.  Oops.

Insurance, anyone?  You do have insurance on your data center don’t you?  Do you know what it covers?  Did anyone update recently?  Does someone have a record of all the servers the sysadmins installed?  Networking equipment?  Storage devices?  What if your insurance doesn’t cover the cost of replacing equipment?  Data loss?  How many millions have you invested in the last couple of years in equipment?  How about your business interruption insurance, is it going to cover your income losses while you rebuild/replace all that equipment, software and employee payroll?  Gee, it doesn’t cover income lost due to IT downtime?  Or worse, it only covers loss when the data center is completely destroyed.  Hmmm, well, sorry you went out of business but some other firm benefitted because your executives decided that having adequate insurance on the data center wasn’t important or cost effective.

There was a data center in that office building?  Oh, that’s a whole other ballgame.  General purpose office space is common.  Raised floor space with uninterruptable power supply, high capacity air coolers, Halon (or equivalent gas based fire suppression system (what idiot put a water sprinkler system in the data center?  Don’t they know electronics and water don’t play well together???)) and network connections, are not common.  It’s one of the reasons, aside from the computerized equipment, that data centers usually have their own recovery plan separate from the building in general.  They are very specialized rooms with very special and time consuming to install requirements.  You can’t just call up a local commercial realtor and rent an office building with all those needs.  For one thing, they are customized to the needs of the electronic equipment they house.  If you lose the data center permanently, figure a minimum of six weeks (heroic efforts required by all the vendors and IT personnel) to get another one configured with network communication access.  Gee, your company went out of business waiting?  Tough luck, at least the exec’s got paid.  You and everybody else will be standing in line at the unemployment office cooking hot dogs over the BC & DR plans you’re using for fuel.

Not what you wanted to hear?  Well, the reality of it is, if you don’t have another data center with all your defined and maintained equipment needs set up and waiting (you can contract with firms who provide rentable data centers but that costs thousands of dollars a month for the subscription) when the data center is lost, go find another job.  It is simply not possible to recover from the loss of everything a data center encompasses unless it’s just one of many the company has.  No real and practical DR plan means no recovery, no ability to do business and eventually, no business.  Oh well, the competition benefitted from your company going out of business due to lack of contingency planning.

The strategies for recovering data centers will be the topic of another article, not this one.

There is another type of building for some companies that I can address but it’s not going to be very helpful.  That’s the manufacturing plant.  Can you recover when that type of building is impacted?  Yes and no.  It depends on the uniqueness of the ‘building’.  In this case, I’m using the term loosely.  We could be talking about a chemical production plant, a building with assembly lines, or even something like a food processing plant.  How do you develop a strategy for recovering those types of plants?

First, check into insurance.  You lose a unique plant you’re only real recovery is to build another one, which costs money.  What you want to look into and help develop; are plans to handle everything up to total loss.  You can’t plan for total loss of that type, so don’t bother to try.  The senior execs will have to decide, at the time, what to do.  Your job will be making sure the life safety programs are in place and tested (evacuation drills, fire alarms, smoke detectors, etc), the  training on handling hazardous material spills is conducted and appropriate(if needed), and printed or written records are recoverable.  Beyond that, let OSHA and the execs dictate what needs to be done.  There are business strategies that executives well above your pay grade can look into but that’s well outside the scope of standard business continuity planning.

Okay, that’s it for this rant, er, discussion.

In the meantime, you’re boss is asking when you’re going to have a plan complete and why don’t you have something to show him and report on and what can he show to his boss?  Well, since your boss isn’t providing anything but demands, as an interim solution until you can get a real plan designed, download some freebie off the internet, fill in the blanks and title it Interim Contingency Plan. What good is it?  It makes your boss happy and gets him off your back while you continue to investigate what is really important to the company and you start to develop a real plan, one that can be of actual value to the business.  Unfortunately, you can do that for your IT department DR plan as well and regretfully, you’ll probably have a better plan then that high priced consultant produced years ago.  It looks impressive, it has lots of pages and you can always put the fact that your produced one on your resume.  (Honestly, it’s a total waste of your time but if it makes your boss happy and they’re paying you, have at it.)

Audited?  What, your plan is going to be audited?  So what?  Either the auditor is going to approve your interim plan because the exec’s don’t want to know it’s useless or they’re going to provide you with teeth to get some real work done.  Either way you’re golden.  Not your fault the audit failed.  You’ve been sending emails and documenting all the negative responses you’ve been getting all along.  You’re not to blame.  You reported to your boss consistently that you weren’t getting what you needed.  It’s his fault that he was more interested in not making waves than he was in protecting the company.  Oh, you’re still going to get blamed?  Well, of course you are.  You’re low man on the totem pole.  You’re damned whether you do or don’t.  Remember, all you can do is collect information and try.  Success depends on the senior executives getting their asses on the ball and insisting that everyone support continuity planning or go find another job.  Until then, be the fly buzzing around that’s not so irritating they kill it (got your resume polished and posted on the Internet?) and keep asking questions.

I’ve gone on about asking questions, so let me detail some of the questions you should be asking.  Over time and with great persistence, you’ve found out what’s important to the company.  One of those things is always the ability to process data. (Really vague statement there, just the kind of wishy-washy generic statement that doesn’t say anything but says everything)  Somebody will pop up, if you haven’t already, and say ‘The data center’!  Ugh.  Yes, we need those computers up and running, those applications available, those customer lists, those big, expensive printers and network access.  All that is encompassed in the meaning of the words: ‘data center’.  So yes, as part of your job (unless the powers that be have separated them) is to produce a DR Plan for the IT Department.  Remember that ‘Interim Plan’?  Remember how many blanks you managed to get filled in and pages printed to fill the 3 ring binder that looked so impressive?  That’s what the IT Department heroes are afraid you want them to produce.  Don’t blame them.  I’ve had the unfortunate privilege of working with book trained consultants who insist on lots of verbiage that’s totally useless in a disaster when the IT folks are frantically trying to bring the production applications back to life.

Back to questions.  You know what’s important (or what’s important to the people you asked) so the next stage is how do you deal with the problem.  A little vague there?  Well, it doesn’t really matter what happened to the building or the data center or the supplier, all that matters is it’s gone.  Whether it burned down or was earthquake damaged or flooded by a broken water main (or a hurricane came through, how dare it!), the point you need to address is, it’s gone.  So what are you going to do?  What do you need?  Who can do it?  How?

When a business decides it needs contingency planning and goes to the trouble of hiring someone to develop their ‘plan’, they rarely have a clear idea of what they mean.  They know they don’t need a ‘plan’ for handling normal, day-to-day, irritations, like server crashes, angry customers or a printer breaking.  Those kinds of things they handle on a local level and only when it gets to the stage of costing the company serious money, does it get noticed higher up.  So, often enough, when you had the job interview with the person doing the hiring, they had vague statements of what they wanted but try to pin them down and all of sudden, that’s your job.  Cool, you get to decide what’s important to the business.  Oh, that’s not what they meant?  That’s what they said.  That’s the practical result of their failure to figure out what’s important.  That’s really what you need to find out.

Great!

Oh, that’s not really what’s going on.  Of course there will be funding available (when you prove a need for something and can justify the expense).  Of course there will be ongoing support staff for anything that really needs it.  Sure the executives really want a good plan written.  They just don’t want to be bothered with it.  Money is tight and resources are stretched thin, so it will have to wait.  After all, it’s not like it important or will make a difference to the bottom line.

If you got the impression I’m a little cynical about how management views continuity planning, you’re right.  Anybody have a different opinion?

I’ve beaten my head against the wall of management indifference since the mid-1980’s and I’m here to tell you, if the senior management in your company hasn’t ever experienced a disaster or had a near brush with one, you probably never will get any support until they are directly and personally impacted by the lack of a decent plan.  So, my advice?  Don’t bother trying to get management to take a real interest in developing a decent, viable plan.  That way lays heartbreak, ulcers and massive amounts of stress.  That doesn’t mean you shouldn’t develop a plan, it means, develop what you can and let the executives take the fall.

How?