Data Centre Memories... was [OT] VERY STUPID RAID QUESTION!

["Ian Lowe" <ian@xxxxxxxxxxxxxxxxxx>]

-----Original Message-----
From: David Buckley [mailto:db@xxxxxxx]
>When they set this stuff up there were (of course) multiple
nameservers, so >this fault couldnt happen, but as time passes, things
change, and suddenly >the world ends.

My favourite was watching one of the local Directors (I was at Crewe
Toll, Edinburgh) throwing a paddy, when they lost the entire datacentre
in one fell swoop - about a week after we had completed the roll out of
super redundant Proliant 7000s, with triple redundant PSUs, twin SMART
controllers, dual failover NICs (to separate switches of course) running
in MCS clusters with each node of the cluster in a different rack,
powered from a different phase on the UPS. *cough*

the Single Monster UPS, that was considered bomb proof as it had ran a
horde of big ICL and IBM kit (including the coolest darn automated tape
library I ever saw!) and was now ticking over with the load of 60 or so
Wintel Servers, and wasn't included in the server reslience study..

So, time passed, and the company that maintained the air conditioning
was removing an old unit outside, the sparks from an angle grinder got
into the ventilation, and the smoke was enough to trip the Halon Purge
system in the datacentre - which failed to fire, instead just sitting
going "clickclickclickclick".

Thirty seconds later, the UPS switched off.. later it emerged that the
fire system was hooked into the UPS, and no-one recalled it ever being
changed when the Big Iron was removed..

Worse, the Halon system was not the new environmentally friendly(er)
stuff, so none of the server jockeys were allowed back in to restart
stuff until the fire brigade had safed the purge system.

Moral I guess - no matter how much money you spend, you will *always* be
exposed to another, even wilder failure that you didn't consider
possible.

I used to work for Standard Life, and got to work in the datacentre at
Tanfield in Edinburgh a few times - it's bomb hardened, built in two
halves, and engineered that at least half of it would survive a 747
crashing into the building - something they considered as a risk many
years before 9/11.

Then again, SLAC's Tanfield DataCentre did cost the best part of 8
milion quid.

I.





UK Home Automation Meet 2004 - BOOK NOW!
http://www.ukha2004.com

http://www.automatedhome.co.uk
Post message: ukha_d@xxxxxxx
Subscribe:  ukha_d-subscribe@xxxxxxx
Unsubscribe:  ukha_d-unsubscribe@xxxxxxx
List owner:  ukha_d-owner@xxxxxxx