[website] Tales of Data Loss: TechReport contest

[sonofsanta]

TechReport have just finished a contest for tales of data loss, and some of the entries are cringe worthy and brilliant: Here are the winners of our Macrium Data Disasters contest - The Tech Report

 

My favourite...

Sunday, 3:40PM. I receive an alert text from a managed client's UPS in the server rack "On Battery". Around 18 minutes later I receive a second alert "UPS batteries discharged". Two minutes later I receive a third alert "UPS batteries discharging". "Good", I thought, "apcd shutdown the file server.. I bet I'll get a phone call tomorrow around 8AM..."

 

Monday 8:15AM, I'm drinking coffee awaiting the inevitable phone call because it always happens. A power outage, shuts down the server, and I have to instruct another temp employee to go to the server room, and press the "amber" power button. Another half hour passes and no phone call is received. "Good, must've figured it out" I think to myself.

 

Around 2PM my phone rings "Hello, this is ChubbyHorse with Data Barn, how can I help you?"

"Yes, the power went out yesterday but the file server is down". I repeat the rehearsed how to guide to press the power button to them.

"Thank you SO VERY much!!!" they say. "You're very welcome; have a wonderful Monday" I say back. ... and that was the end of that. Or so I thought.

 

Tuesday, 3:12PM my phone rings; it's my client "Um, yeah, the file server is down again"

"It is? What is it doing?" I ask.

"It's stuck rebooting.. and it keeps saying press F1 to continue or F2 to accept data loss".

"Oh that's not good; I'll be right out."

I arrived at their office, and checked in, and was soon escorted to the server room where I met my victim. After a little poking around, I confirmed the system had a hard drive failure; but was curious why it was acting the way it was. 8 drive raid 6 with online spare I shouldn't have this issue. More digging around, and you guessed it, three drives failed. The E400 raid controller didn't have enough drives to continue, and after looking through the system logs, the last drive failed during rebuild too. "Well, this is going to be a late evening rebuilding this server and reloading the data" I thought.

That's when I noticed the external HDD I installed was missing. I asked where it went, and was told one of the office employee's took it because the owner wanted one for his computer, and figured this one wasn't doing much sitting plugged into the server. *face palm* What about the errors that were getting emailed to you guys?

Well a former office employee knew just enough to be dangerous with Linux, and, well, long story short, took the back up drive on the server, and re-purposed it. Got tired of the errors getting emailed of the failing backup so they deleted the script file. Then got tired of getting an email from cron (Linux task scheduler) stating it couldn't find a file every night; so they disabled the xinetd daemon to stop cron; which killed the SMART monitor program, and the off-site sync program. They later, found the "google drive" stuff all old, and so the deleted it so the old data wouldn't get used by accident.

*My heart stopped*

500GB of financial, SOPs, pictures, contracts, CAD drawings, audits. Gone.

 

"There is nothing I can do for you" I said.

"So it's all gone?"

"Yeah"

 

They were not happy office employees; nor was the owner happy. I thought for a minute they were going to go office space on the poor old Proliant. I even had a bit of remorse for them because I was there for a day and a half setting the server back up, and getting QuickBooks to play nice again, and oh boy the commentary I heard. I ended up only charging for a 1/4 of my time because I couldn't add more insult to injury. Thought, I think they blamed me however because I was the tech guy who installed everything once upon a time and I never got called back after a follow up call the following week.

[bald_pig]

Eesh, something to browse between tickets, thanks!