Apparently, the chances of winning the lottery are considerably more slender than the chances of being a plane crash victim. So, if you buy lottery tickets, a salutary lesson in risk management might be a good idea!
Kim Thomas finds that technology has plenty to offer in keeping essential public services running when the lights go out.
Two years ago, Wakefield Metropolitan District Council experienced a brief power cut – the kind of mundane interruption to business that happens from time to time. But when the power came back on after two hours, it spiked. “The uninterruptible power supply was fine,” says Alan Kirkham, service director for procurement and e-services at Wakefield. “But the spike killed the air conditioning and the backup air conditioning, so the data centres overheated tremendously. We lost our main Exchange email server for about three days before we got everything restored.”
Most of us, when we think about business continuity problems, tend to imagine the biggies: floods, terrorist attacks and fires. More often, however, it’s the small, everyday disasters that cause the problems: the entire office comes down with flu; the server is attacked by a new computer virus; or, as in Wakefield’s case, there’s a power cut.
Remember Scouts and Guides? Be prepared!
Top ^
Any organisation, whether it’s a business or a local council, needs to have a strategy for dealing with risk. The initial risk analysis needs to consider what the individual interruptions might be, the likelihood of them happening, and the likely impact of an outage on customers.
Just as it’s easy to forget the mundane one-off risks, it’s easy to forget the “continuous” risks, such as the failure of a member of the supply chain, says Graham Titterington, principal analyst at Ovum: “If one of your key suppliers fails and you haven’t got an immediate fallback lined up, then your processes come to a standstill just as much as if someone has bombed your office.” The risk increases at a time of economic recession, and applies just as much to the public sector as to business, he adds.
Many risk analyses also make the mistake of failing to take into account the probability of more than one adverse event striking at the same time, says Titterington: “Most of the things [in a risk analysis] are relatively unlikely. You think of all these unlikely things and think, ‘We’ve put in place some sort of plan to deal with them when they happen, so we’re OK.’ We assume that because they’re unlikely, the chance of two of the things happening is minuscule, but sometimes two unlikely events might be triggered by a common cause. The chances of the second one happening once the first has happened are quite high. And if your fallback procedure relies on the second one, that’s not a very good policy.”
As an example, Titterington cites the incident in 2002 when a JCB digging up the road in central Manchester cut through a broadband cable. Not only did many organisations find themselves offline for a fortnight, but mobile phones no longer worked because the mobile transmitters had been cut off, too.
You can guard against this by taking into account the possible knock-on effects of an interruption to service, such as an increase in demand elsewhere. “When something goes down, the demand for alternative service drastically increases, and sometimes services are configured in such a way that excess demand can cause total failure,” says Titterington. This happened during the North America power cut in 2003: the reserve routes on the grid were knocked out by the repercussions of the first failing.
Business Continuity in action
Top ^
The aim of business continuity planning is to be able to restore service to normal as quickly as possible after an interruption. If the data backup is also affected by the interruption, then that will be difficult to achieve, so good practice is to make sure that the backup isn’t too close geographically to the original data.
When Wakefield Metropolitan District Council decided to upgrade from Microsoft Exchange Server 2000 to Microsoft Exchange Server 2007, it also adopted Microsoft System Center Data Protection Manager 2007 to help with backup and recovery, and made the astute decision to deploy essential servers across two sites with failover, rather than one. “We replicated our main Exchange servers on two different sites about 11 miles apart, with a gigabyte connection between the two,” says Kirkham.
The decision meant that in November 2008, the council was prepared when a power cut hit Pontefract, where one of the servers is located: “A 26-hour power cut hit the whole town, and absolutely no-one else in the council knew about it. The business continuity side kicked in perfectly,” says Kirkham. “We had no mains power in the whole building, but everything in the building just continued and all the services carried on normally. With our Exchange Server configuration, if we were to have a crash on one server, the other one would kick in automatically.”
Work from home capability
Top ^
It’s also important to bear in mind that, in some situations (the Buncefield explosion, for example), employees will not be able to come into the office. Titterington advises taking steps to enable as many people as possible to work from home: “That has the advantage of being a distributed and therefore robust option, so providing them with the IT and telecoms infrastructure they would need in that eventuality is a good precaution.” To support that strategy, it is worth considering a hosted solution, such as Microsoft Exchange Hosted Continuity, which provides continuous email access if the primary email system fails. It means that even if employees can’t get into the office, then they can access email from home via the web with minimal interruption to service.
It’s unlikely that your organisation will be able to prepare for every eventuality, but any contingency planning should take into account the fact that the chances of business being interrupted by ordinary events such as power cuts and flu epidemics are quite high. As Titterington says: “There is a problem in the human consciousness of over-reacting to the spectacular but unlikely, but under-reacting to the more mundane but fairly serious.”
Don’t forget…
- Business continuity isn’t just about planning for terrorist attacks – you need to plan for mundane events such as power cuts.
- Two events can be triggered by a common cause, so make sure your backup plan is as self-contained as possible.
- In situations where employees can’t get into the office, then make it possible for them to work from home to minimise disruption.
If one of your key suppliers fails and you haven’t got an immediate fallback lined up, then your processes come to a standstill just as much as if someone has bombed your office.
Graham Titterington
Ovum
A 26-hour power cut hit the whole town, and absolutely no-one else in the council knew about it. The business continuity side kicked in perfectly. We had no mains power in the whole building, but everything in the building just continued and all the services carried on normally.
Alan Kirkham
Wakefield Metropolitan District Council
Kim Thomas is a freelance journalist, who specialises in writing about technology, business and education. Her clients include the Financial Times, the Economist Intelligence Unit and The Guardian as well as a number of B2B publications.