[mattx]

by Alex Papadimoulis

A few months back, G.R.G. shared a story about the “Secured” Server Room at a certain university he once worked at. Like so many WTF’s, there’s a fun follow-up to this story. Brief recap: the dedicated air conditioning unit for the “ultra-secure” data center had died and none of the elite group of key holders could be reached to open the door; fortunately, G.R.G. was able to unlock it with his trusty Bic pen and let the maintenance guy in to fix the problem.

All the servers, all the research grants, G.R.G. thought while basking in his Bic-powered breaking-and-entering skills, I could take them, they’d be mine, mine, mine! As G.R.G. briefly pondered upgrading to burglary, the Maintenance Guy hastily worked to vent the sweltering air from the room. A few minute later, with the aide of a giant floor fan, the temperature dropped to a much more suitable level.

While G.R.G. worked on rebooting the downed servers, the Maintenance Guy inspected the nonfunctioning A/C unit. “Aww crap,” he said after cycling the breakers a few times, “I bet it’s those damn leaves again cloggin’ up the compressor unit.” He told G.R.G. that he’d have to go up on the roof and inspect things from there.

One thing to keep in mind about this certain university, is that it’s located within in a certain geography known for its rapid transitions from Winter to Spring. On April 5th, there’s snow on the ground and the mercury reads a brisk 20°; on April 6th, it’s a cool 70° and everyone’s walking around in shorts. And it’s usually on this day that the dedicated A/C unit fires up.

Just as G.R.G. was finishing rebooting the servers, the Maintenance Guy returned from the roof. “Yup,” he said casually, “it’s the leaves again. The whole A/C unit’s shot until we can get it cleaned out.” A day or so of A/C and server repairs, and the datacenter was back up and running.

The Spring months passed by and then, all of a sudden, it was Summer. The A/C kept on cooling through the blistering heat and all the way throughout the Fall. Sometime in late October, as expected, the A/C shut down and entered its several-month hibernation. The poor insulation and frigid winter months was all the cool the server room needed.

And then, all of a sudden, April 6th hit again. A Maintenance Guy rushed to the building in response to a temperature alarm, frantically ran around trying to find someone to open the door, and eventually ended up on the roof with the same diagnosis: “the A/C is clogged up with leaves, again.” After a panicked day or two of no cool, the A/C and servers were patched up and working again.

Inquiring further with the Maintenance Guy and the datacenter administrators, G.R.G. learned that this whole routine had become a bit of a tradition at the university. On or about April 6th – whenever that first blast of Spring weather hit – the A/C system would fire up and then quickly grind to a halt due to leaves. It had been going on for as long as anyone could remember, and everyone had assumed that air conditioners just work that way.

As G.R.G. was leaving the building that day, he glanced up at the roofline. For the first time, he noticed that, when that old building was remodeled, the contractors added an architecturally-compatible brick cooling tower for the air conditioner. And perched directly above the cooling tower was a majestic, 200-year-old ash tree. It was perfectly positioned so that, every year, at least half of the tree’s leaves would fall directly into the cooling tower.

Figuring that, somehow, the datacenter admins or A/C repair folks missed the painfully obvious fact that a quick brushing-off the leaves before starting the unit – something that could be done once, any time between November and April – would save thousands of dollars in equipment repair, a bunch of man hours, and whole lot of lost productivity time, G.R.G. sent a friendly email to the maintenance and datacenter managers. “Thanks, but we know what we’re doing,” about summed up the response.

Like clockwork, on or about the following April 6th, A/C unit died from clogged leaves again. G.R.G. left that job shortly thereafter, but is pretty sure the tradition continues to this day…