Data Centre meets Vacuum Cleaner

Hi Folks,

I have just spent a couple of days resolving some problems at the remote DR data centre, sorting out the problems caused by the over zealous use of a Vacuum cleaner of all things.

We have a backup server a SUN V480R with a Storedge 3510 and expansion attached which suffered a significant unexplained failure, all tracked back to an ID selector being touched by the nozzle of said vacuum cleaner - it looks like things went as follows over a period of time.

When the array was installed the setup was disks 0-9 were setup as a 10way stripe with disks 10 and 11 as hot standby disks. Over a period of time, a disk in the expansion (disk 3) failed and the and the first available spare (disk 10) built from the surviving mirror.

At this point the situation that existed left us exposed in a way which wasn't really appreciated, in that the one of the arrays had both mirrors of one part of the stripe. That would be the part that had the exposed ID selector switch, the one that the Vacuum Cleaner nozzle could change causing the failure of one whole stripe and one slice of an other stripe. The result as you can imagine was somewhat unpredictable, which is exactly what the Sun manual for the array says.

To add insult to injury the contents of the 3510 array, was the Legato Networker Backup Catalogue from the 24 drive ATL - making the recovery somewhat awkward.

What's the point of the story - don't let some idiot into a data centre with a Vacuum Cleaner.

Regards

Gull04

Or, moral of the story, make sure buttons have the cover in place to prevent accidental bumping.

Hi Joeyg,

You are of course right, however I was advised that this solution was built from disposed of equipment as a short term solution in 2006 and the cover for the ID switch was left in the skip.

Not much help when you are performing the headless chicken dance in the data centre I know, but I'm told that ensuring this won't happen again is now a priority.

I think that they'll probably try and get a cover off aBay, this will probably mean that it will be in service as a temporary solution for an other 12 years.

Regards

Gull04

I had a similar experience a few years ago when an additional storage unit was urgently added to a system. The cleaner entered the highly secure computer room and unplugged the new unit to plug in her vacuum cleaner. When challenged whether she had switched off the new storage she simply said, "Well yes, I've always used that socket".

Many years ago, while I was working at Sun Microsystems, Inc. on adding POSIX-conformance into SunOS 4.1, I was making changes to the OS and utilities during the day and running complete builds of the system overnight (starting a build just before I left work in the evening). This worked fine for several weeks until one Monday night when the system died at about 8:30pm killing the build-in-progress. There was no core dump, no indication of any hardware problems, and restarting the build when I got to work Tuesday morning completed normally (taking a little over three hours to complete).

The same thing happened the next three days in a row, with the system always dying sometime between 8:20pm and 8:40pm.

I decided to stay at work late Friday evening to see if I could figure out what was causing the crashes. I went to the bathroom at about 7pm so I would be sure that I could be at my computer by 7:15pm and would be able to stay there until I found out why my computer was dying every night. When I got back to my office five minutes later, I found that my computer had been unplugged by a member of the cleaning crew so he could plug in the vacuum cleaner they used to clean the hallway and offices where my office was located. (The way my office was arranged left a power strip close to the hallway while other offices in my area had their power strips in less accessible locations.) I unplugged the sweeper, plugged my computer back in, and waited for the cleaning guy to come back to my office.

I found out that:

1. the cleaning crew comes in an hour and a half earlier on Friday that they did Monday through Thursday,
2. they got a new vacuum cleaner on Monday with a cord that wouldn't reach from the plugs in the conference room to the other end of the building, and
3. unplugging a computer is inconsequential to a janitor if doing so allows him to sweep the carpets at the end of the hallway.

It's not a good idea to call cleaners and others "idiots" as their job is not to develop IT policy or data center cleaning procedures. Their job is to follow procedures for how to clean a data center and to be supervised when cleaning, especially if the data center has critical systems.

Hence, the problem is not "the cleaner" but the management and the administrators of the data center who seemingly did not put proper cleaning procedures, controls and supervision (if required) in place to insure that anyone who worked in the data center understood the proper procedures on how to clean and vacuum in a data center.

Even in my home, I don't permit our biweekly housekeeper to clean and vacuum around my key IT devices unless I am there.

So, for a data center with critical data and business systems, the responsibility falls on the data center management and administrators, not the "cleaners," to insure all personnel working in the data center understand how to do their job in a careful and responsible way.

FYI, forum rules do state:

When we founded these forums decades ago, we have always been careful to insure that everyone is treated with respect (unlike many other site, where anything goes and name calling is normal); and although we may disagree, we do not call each other names.

We would never call my housekeeper a bad name, for example. She is nearly blind and cleans the mirrors and glass with her face a few inches away, and if she needs to be supervised, we give her gentle instructions.

I have worked in many data centers, some very critical and highly secure. Data centers should have clear procedures for how to clean, how to vacuum, where to plug in appliances, how close to get to the electrical gear and more.

It is typically the fault of data center managers when these types of incidents happen; as most are simply too distracted or not "very detailed oriented" to create housekeeping procedures that insure safe housekeeping operations in a data center.

Honestly, who in their "right data center management mind," who runs a data center, would have a cleaning person or team just "plug in" wherever they wanted or to clean near wires and racks without clear procedures and supervision if necessary?

Yes, and to have proper procedures on how to clean, where to plugin, how close to get to wires, gear, etc. and to have proper supervision if the data center has critical servers that perform critical business tasks and functions.

We call these "controls"..... there are "physical controls" (button covers, LOL), "technical controls" (backup UPS systems, etc) and "administrative controls" (procedures, supervision... )

It's all pretty basic "IT security"