Kings College London SAN failure and data loss post mortem

[mjk]

This hadn't ever occurred to me, this sounds like a brilliant idea. How did you set the email up?

There is a neat hack for Google apps users to do this, using the email report log generated by your backup app:

 

Automated email-to-task mechanism with Google Apps Script | DJ's Weblog

[penfold]

@AButters 7 @foofighterjim We use Spiceworks yes. I have a server with a scheduled task which sends an email every morning into the Spiceworks email account. I've set it up with the appropriate header "Daily Systems Check" and use Spiceworks rules to assign it to the appropriate category and set Due date and auto apply the Checklist. I've also got it cc'ing other staff so they can see the job come in. Only thing then is to mark the checklist off as you go through the checks.

 

Edit: The generation of the email has been done through Powershell and run using Task Scheduler.

 

Edit 2: I was hoping Spiceworks could automate it within the helpdesk but searching the forums it seems like the only real option was to schedule an email going into Spiceworks.

Edited March 1, 2017 by penfold

[psydii]

We just get the backup system to email the service desk directly. The job is not auto allocated so if the nominated person is off, another member of the team will see it.

 

The service desk system parses the email for keywords, and escalates if the backup system is reporting errors or warnings - a warning or error generates a Problem ticket.

 

Incident/Problem/Change ticket stats are reviewed by SLT monthly and the Head quarterly. The report is set to filter out business as usual 'backup success' tickets, so only those days where there has been some sort of problem appear.

 

It still is prone to some of the same mistakes KCL made though:

What if the backup system stopped emailing - and for what ever reason the nominated person didn't notice?

What if the backup system was bombarding the service desk with 'warnings' which were not relevant (who would make that call)?

What if then suddenly some warnings became relevant?

What if Senior Management did not effectively question backup errors or warnings, or IT Management did not communicate the urgency?

[Davit2005]

We just get the backup system to email the service desk directly. The job is not auto allocated so if the nominated person is off, another member of the team will see it.

 

The service desk system parses the email for keywords, and escalates if the backup system is reporting errors or warnings - a warning or error generates a Problem ticket.

 

Incident/Problem/Change ticket stats are reviewed by SLT monthly and the Head quarterly. The report is set to filter out business as usual 'backup success' tickets, so only those days where there has been some sort of problem appear.

 

It still is prone to some of the same mistakes KCL made though:

What if the backup system stopped emailing - and for what ever reason the nominated person didn't notice?

What if the backup system was bombarding the service desk with 'warnings' which were not relevant (who would make that call)?

What if then suddenly some warnings became relevant?

What if Senior Management did not effectively question backup errors or warnings, or IT Management did not communicate the urgency?

 

That's why IMO you should as much have a success notification as you do a failed notification. The team responsible should all get emails, if their is only one tech in the team then the person he direct reports to should also get emails. The responsibility should be on one person to check backups but there should always be a 2nd/3rd person who WILL pick up if the responsible nominated person is un-available.

 

At the last place I worked in I was initially made responsible and made sure the systems were updated and issues dealt with but the weekly tape swap over was given to another Senior Systems engineer. I picked up the tape rotation tasks automatically when he was not available, if neither of us were available then my line manager picked it up.

[mavhc]

Backup is a useless word, do you want to recover from a) someone overwriting/deleting a file: snapshots; b) hardware failure: replication (drive failure: raid, computer failure: to another computer, building failure: offsite)

 

Also c) archiving.

 

Next question: you're altering x GBs/day, backup space costs £y/GB, how many copies do you want to store, and can you afford that?

[localzuk]

I always think of backups in 2 ways - day to day, and disaster recovery. Archiving can be an extension of either - depending on the reason for archiving. Eg. We create an "archive" on a yearly basis, per our finance policy, for audit purposes - all our data is archived onto some form of media, and stored in a third party site, dated the day after our financial year ends. That way we have a perfect copy of our finances should auditors need them.

Any backup solution should cover all, whilst not needing entirely separate solutions.

Edited March 1, 2017 by localzuk

[nicholab]

I thought about this you got servers that you only want to be able to go back a month then sims/fms and fileserver that you want to go back ages on.

[Davit2005]

Backup is a useless word, do you want to recover from a) someone overwriting/deleting a file: snapshots; b) hardware failure: replication (drive failure: raid, computer failure: to another computer, building failure: offsite)

 

Also c) archiving.

 

Next question: you're altering x GBs/day, backup space costs £y/GB, how many copies do you want to store, and can you afford that?

 

If xGB data is not backed up due to bandwidth, cost, technical, etc.. issues then SMT must be made aware of this. It is their call if some SIMs data or historical email is needed for investigation. Not sure about Archive, this is a backup no????? Is their any need to keep data over 15 years old in your organisation.

 

Shadow copies can be used for end users who need to restore files, recover emails used in case some one accidentally deletes an email. A backup to restore an SQL table, mailbox, corrupt/ransomeware'd file system, a corrupt server, DR, etc.......

 

 

I thought about this you got servers that you only want to be able to go back a month then sims/fms and fileserver that you want to go back ages on.

 

Different jobs with different retention periods/different media sets etc, every backup job should be capable of that IMO. Even my Synology has different retention settings for backing up data.

[mavhc]

If xGB data is not backed up due to bandwidth, cost, technical, etc.. issues then SMT must be made aware of this. It is their call if some SIMs data or historical email is needed for investigation. Not sure about Archive, this is a backup no????? Is their any need to keep data over 15 years old in your organisation.

 

Shadow copies can be used for end users who need to restore files, recover emails used in case some one accidentally deletes an email. A backup to restore an SQL table, mailbox, corrupt/ransomeware'd file system, a corrupt server, DR, etc.......

 

 

 

 

Different jobs with different retention periods/different media sets etc, every backup job should be capable of that IMO. Even my Synology has different retention settings for backing up data.

 

Right, need some kind of dashboard: Department x is paying £y to backup z GBs of data, this excludes: a (would cost £a') ,b (£b'), c,d,e,f... Last backup completed at $date