On a chilly spring afternoon, IBM's western regional manager invited us to bid on a most challenging project: "Due to new earthquake guidelines we need to upgrade our most critical data center floor, panels and struts... a shutdown will cost millions."
I was nonplussed, figuring that a dozen or so racks was no real challenge. We have often engineered complex logistical solutions, with secure mirrors and rapid deployments. But, when Scott told me that the center was comprised of over 400 fully stocked IBM, EMC, Dell, and HP enclosures he had my full attention.
Failure Not an Option
He explained that these enclosures managed both the active electrical grid and service for a few million customers in multiple western states. A failure would most likely result in rolling blackouts, lawsuits, and very angry customers.
He was putting the project out for bid, and he had come to us on the recommendation of a large ISP whose operations we had relocated a few months ago. We won the bid after some major internecine battles, which culminated with a 76 person conference call between 75 sharp stakeholders and yours truly.
We considered many options. Traditional relocation and virtual accommodation methods were impractical due to both the politics and key operational concerns. Jointly we decided on an innovative vertical solution.
Vertical methods, wherein the equipment would be lifted off the raised floor panels by a minimum of 2 feet, and held for 2 hours, was needless to say controversial. However, we quickly discovered that the lifting of the equipment was only a small portion of our challenge.
I recalled the sage advice that "all tasks are fundamentally simple... it's the quality of solutions to the inevitable sharp problems that occur, that is the truest test of success."
The primary challenges were in both the logistics of minutia (such as cable strain and interface integrity management) and the larger issues of stabilizing active enclosures, with drives spinning, and rack cooling fans whirling.
Once the project got started, it took 2 months just to figure out the logistics, and to get the equipment that we needed designed and manufactured. After a few fits and starts, some of the appliances that we had developed proved untenable due to weight distribution and center of gravity issues which we discovered during exhaustive pre-test simulations.
Stability and Safety First
There were of course no shortage of ideas from our engineers. Everything from overhead cranes to hydraulic platforms were proposed and tested. We finally decided on a series of procedures that provided maximum stability, and extraordinary safety. A crew of 6 headed by a senior structural engineer, required from 4-8 hours per enclosure to accomplish the tasks safely.
Each phase of the six-phase operation required an extensive checklist. These checklists embodied situational and procedural guidance for the crew, while simultaneously generating documentation for OSHA, the insurance companies, and the ever nervous stakeholders.
Success in the End
We will skip the minute details for now, but we can report complete success.
We successfully performed over 400 vertical relocations, over a 6 month period... without a single outage. The utility's customers never lost power, their bills remained accurate and timely, and none of the stakeholders jumped, as threatened, from high office windows.
A carefully executed, ego-less, yet simple plan is the stuff of success.
Author Gare Henderson is the V.P. of operations of High Technology Equipment Movers, Inc., established in 1989.