e i g h t - c u b e d . c o m 

Blackout

Last night, I was watching TV after dinner. This is unusual for me unless there is a good movie on, but I wanted to see how much of a propaganda vehicle "The Grid" is. I didn't get to find out, because about 20 minutes into the show, the power went out.

After worrying about my computers, and disconnecting them as a precaution for when power was restored, I went to bed as it appeared obvious that we would be blacked out for an extended period.

Twenty minutes later, my phone was ringing, and someone was telling me the disturbing news that the data centre had completely lost power.

The company that employs me has an extensive data centre. It's fully protected by a large UPS and a huge diesel generator that has enough fuel to run for a week without being topped up. Or so we thought. Obviously, it failed to do its job.

Unfortunately, I have experience with reviving this site from a complete power failure as I've had to do it before when a contractor mistakenly hit the emergency power off button.

Management wanted an estimate of how long it would take to get everything up and running. I said nine hours, as that is about how long it took last time. The sad thing is that I knew the answer.

The pressure was on because Operations needed to run the primary application's backup and batch run before the online system could be brought up in the morning. With me concentrating on the production system, and the other systems people working on the data warehouse and other peripheral systems, we had everything up and running by about 4 AM.

Backups and batch were run, and the online system was up at 8 AM, only an hour late.

I'd like to say "thanks!" to the rest of the VMS team. You all pulled together nicely to get everything back up and running without complaint and made it easy for me to focus on the production cluster. Give yourselves a pat on the back.

And to the boys in Facilities, sort that UPS out for us, please? I have enough grey hair already.