Fire Safe Backups...

[CHiLL]

At the moment, we have limited disaster recovery and I am looking at implementing an improve that. Our main backups are performed by VEEAM which backs up all our VMs to a SAN (D2D/Disk to Disk). I'd like to copy these backups onto other disk drives which I can safely store in a fire safe (D2D2D).

 

We have approximately 3TB of data to transfer to the fire safe disks each day and we have a small Netgear NAS box which is filled with 6x 3TB drives configured as one large 18TB (RAID0?). I could remove some disks leaving 2x 3TB disks in...but do I have them in a RAID0 setup or do I have them appear as individual disks?

 

I've seen an issue in a previous job, where we were required to retrieve data from an End of Year backup from a few years previously. This data was stored on a single disk, which failed during its time in the fire safe. I'd like to avoid this sort of issue is possible.

 

Another concern I have is regarding the RAID setup...if the NAS box fails...how would I retrieve data? As the NAS model we have was discontinued some time ago.

 

I know this could spiral out of control when attempting to prevent failures, etc...but can anyone provide any advice?

[Michael]

Working on the presumption you have a 100Mbps BGFL connection, this would take 70 hours to transfer 3TB of data, so a cloud backup is unrealistic (if you want full backups), but incremental cloud backups could be one option.

 

As for the RAID setup on that NAS, you need to implement RAID5 giving you 12TB of space, but more crucially redundancy. Obviously the only other issue is how would you store this in a fire proof safe. The other solution are of course backup tapes. Sony have developed a tape that can hold 185TB, but as to how much they are, I've no idea but it goes to show what's possible.

[CHiLL]

Working on the presumption you have a 100Mbps BGFL connection, this would take 70 hours to transfer 3TB of data, so a cloud backup is unrealistic (if you want full backups), but incremental cloud backups could be one option.

 

As for the RAID setup on that NAS, you need to implement RAID5 giving you 12TB of space, but more crucially redundancy. Obviously the only other issue is how would you store this in a fire proof safe. The other solution are of course backup tapes. Sony have developed a tape that can hold 185TB, but as to how much they are, I've no idea but it goes to show what's possible.

We wouldn't be doing it over the cloud, the backups would be held on-site, in fire safes, in different physical buildings.

 

I don't think tape is an option.

 

Using RAID5...what if the NAS box died? I thought the RAID setup was specific for that hardware?

[Arthur]

This data was stored on a single disk, which failed during its time in the fire safe. I'd like to avoid this sort of issue is possible.

The easiest solution would be to purchase an ioSafe (which is a fireproof and waterproof NAS).

[pleach85]

We copy our nightly backups to an encrypted RDX drive which we then place into a fire proof safe. As far as I am aware RDX drives only go up to 3TB at the moment so this might not be suitable although they do a multi drive system called an RDX QuikStation which might be worth looking into. Granted it's still a hard drive (at the 3TB size) but they're built to last.

[twin--turbo]

Still use tape here for long term backup. Ok it's done by an autoloader at the other site across the WAN but i dont see why a single LTO6 direct attached to your backup server is not a viable option for you!

[Michael]

We wouldn't be doing it over the cloud, the backups would be held on-site, in fire safes, in different physical buildings.

 

I don't think tape is an option.

 

Using RAID5...what if the NAS box died? I thought the RAID setup was specific for that hardware?

 

But that's like saying what if your motherboard (in your server) died. RAID5 is more appropriate than RAID0 with no redundancy. It's impossible to predict or control every scenario, but it's clear you want to do everything possible which is great

[Davit2005]

Have you thought about copying the files to a removable drive such as RDX, removable drives such as USB drives etc are designed to be removable although yes you can hot swap drives in nearly all NAS devices but they are not really designed for that purpose in my view, other people may differ.

 

For home I use RDX but at work we have a LTO6 autoloader.

[CHiLL]

Have you thought about copying the files to a removable drive such as RDX, removable drives such as USB drives etc are designed to be removable although yes you can hot swap drives in nearly all NAS devices but they are not really designed for that purpose in my view, other people may differ.

 

For home I use RDX but at work we have a LTO6 autoloader.

 

The idea was:

1) Have three sets of HDDs (1x active set, 1x in fire safe 1, 1x in fire safe 2)

2) Fit set 1 HDDs in NAS

3) Turn on NAS

4) Copy data to NAS

5) Once copy completed - turn of NAS box

6) Remove set 1 HDDs

7) Store HDDs in fire safe 1

8) Fit set 2 HDDs in NAS

9) Repeat from step 3)

[Michael]

I've never considered using a NAS in that way. The other issue of course is you need to remember to put them in the correct order for the RAID configuration.

 

Forgive me for asking, but it sounds like you're managing a big school, so the odds are it has separate buildings? Putting a NAS in each would probably be just as reliable and secure. You also have to consider a lot of schools are surrounded by houses, so in a way they also would notify the authorities in the event of a fire out of hours, if just to protect their own property!

[Davit2005]

The idea was:

1) Have three sets of HDDs (1x active set, 1x in fire safe 1, 1x in fire safe 2)

2) Fit set 1 HDDs in NAS

3) Turn on NAS

4) Copy data to NAS

5) Once copy completed - turn of NAS box

6) Remove set 1 HDDs

7) Store HDDs in fire safe 1

8) Fit set 2 HDDs in NAS

9) Repeat from step 3)

 

The other thing you have to think about is what happens in 2 years when the NAS has an issue.

 

At least with other types of removable drive backup devices they are fairly interchangable and you can move them to another server wether they be SAS, SCSI, USB or SATA interfaces. You don't have to worry about importing a foreign raid set of drives into different hardware.

Edited March 14, 2016 by Davit2005

[Michael]

I agree - if I were backing up 3TB of data, I'd look to have several strategically placed NASes around school, not just one, despite the ever increasing sizes of hard drives making things easier.

 

At some schools I already do this, despite not backing up anywhere near 3TB of data. It's not expensive when you weigh up the benefits.

[CHiLL]

I've never considered using a NAS in that way. The other issue of course is you need to remember to put them in the correct order for the RAID configuration.

 

Forgive me for asking, but it sounds like you're managing a big school, so the odds are it has separate buildings? Putting a NAS in each would probably be just as reliable and secure. You also have to consider a lot of schools are surrounded by houses, so in a way they also would notify the authorities in the event of a fire out of hours, if just to protect their own property!

It's not a large school, though it does have two separate buildings. That is an idea, but I'm hoping to persuade the school to buy fire safes.

 

The other thing you have to think about is what happens in 2 years when the NAS has an issue.

 

At least with other types of removable drive backup devices they are fairly interchangable and you can move them to another server wether they be SAS, SCSI, USB or SATA interfaces. You don't have to worry about importing a foreign raid set of drives into different hardware.

That's one of the concerns I mentioned earlier. The NAS usage idea was taken from the last school I worked in. They had their HDDs labelled so they inserted in the correct order.

 

Removable media sounds like the next best option in my opinion.

 

I agree - if I were backing up 3TB of data, I'd look to have several strategically placed NASes around school, not just one, despite the ever increasing sizes of hard drives making things easier.

 

At some schools I already do this, despite not backing up anywhere near 3TB of data. It's not expensive when you weigh up the benefits.

I thought that the school had a legal obligation to protect the data from such events as a fire?

Edited March 14, 2016 by CHiLL

[Davit2005]

I agree - if I were backing up 3TB of data, I'd look to have several strategically placed NASes around school, not just one, despite the ever increasing sizes of hard drives making things easier.

 

At some schools I already do this, despite not backing up anywhere near 3TB of data. It's not expensive when you weigh up the benefits.

 

I done this at a previous last place as there were also 3 staff living on the site, for long term I also used tape though. Belts and Braces :-) .

[Michael]

I guess the other advantage (being mobile like myself), is I can restore data remotely. I don't need to be on site, no matter where I'm located that day.

 

Remember each NAS 'should' also be RAIDed as a precaution, be it RAID1 or RAID5 is normally suffice.

[dhicks]

Using RAID5...what if the NAS box died? I thought the RAID setup was specific for that hardware?

 

If you use a NAS that uses Linux's software RAID (MD RAID) you should be able to move the disks around between different hardware with no problem.

[CHiLL]

If you use a NAS that uses Linux's software RAID (MD RAID) you should be able to move the disks around between different hardware with no problem.

Hmm, didn't know that! I was hoping to use FreeNAS.

[dhicks]

I was hoping to use FreeNAS.

 

FreeNAS is based on ZFS. I imagine it should be quite possible to take apart a ZFS volume and put it back together again on different hardware, but you might want to test that before you go relying on it.

[Arthur]

I was hoping to use FreeNAS.

With ZFS you can easily replicate the data to other ZFS systems and it's faster than rsync.

 

http://arstechnica.co.uk/information-technology/2015/12/rsync-net-zfs-replication-to-the-cloud-is-finally-here-and-its-fast

 

Have you considered setting up two NASs and then replicating the data between them automatically?

Edited March 15, 2016 by Arthur

[TechMonkey]

Don't forget a fire safe isn't infallible. They are normally rated for so long at such a heat. The idea is that hopefully before the fire safe heats up the fire brigade has gotten there and started damping the area down to cool the safe. I would say the the redundant remote NAS boxes would be a better idea. Your system sounds quite ingenious but also adding in a lot of complexity and elements that could go wrong.

 

The only other option I can think of was we used to have a Seagate Free Agent system. Not sure if they are made but there must be something similar around. This was an adapter plate that external disc boxes would plug into. We have some 3TB ones so they may do larger. Our VEEAM set-up would copy the backups to it and then we would swap for the next day. We stopped using it as having a remote NAS just meant we didn't have to worry about swapping and less chance of forgetting or loosing the box.

 

That's the other consideration, if you are taking the backups off site: Are they encrypted? Are you insured? Does the school insurance cover them off site?

[CHiLL]

Thanks for all of the replies. I have decided to scrap the fire safe idea and gone for another two NAS boxes in two other separate buildings. However I'd wondering what the best way to transfer our VEEAM backups to these NAS boxes is. So far I can only see two options:

 

1) Use VEEAM's Backup Copy job

2) Use Robocopy

 

I'd prefer to keep it all within VEEAM if I can, however VEEAM's scheduling options for Backup Copy jobs appears to be limited. For our normal backup jobs we have one job per server, so 19 jobs in total, with the first server scheduled to backup at 9pm, and the rest scheduled to start once the one before has completed. I'd have liked to replicate this within the backup copy jobs, for simplicity sake. However when a backup copy job is created, the only scheduling option is to allocate a time slot, so it is not as comprehensive scheduling backup jobs. I could create a job containing all the backup jobs, but I was hoping to avoid that if possible.

 

I can create a Robocopy job, but I'd like to have email notifications on failure, like VEEAM sends.

 

Anybody have any ideas?

[Steve21]

Only way I could think of doing that is Having a Script per Job that runs the backup copy job. (There's option in advanced settings to run script after a job finishes)

 

Steve

[TechMonkey]

We use the Copy jobs throughout the day. I'll take a look at the settings as I don't touch them often. Only use a copy to a single remote space so not sure if you will have to set-up 2 jobs or be able to copy to 2 locations with one job.

 

EDIT: Took a look, it is a VMWARE Backup Copy. The main backup job is selected as the object to process, it is a direct job and the Schedule is Any Time.

Edited March 21, 2016 by TechMonkey

[CHiLL]

We use the Copy jobs throughout the day. I'll take a look at the settings as I don't touch them often. Only use a copy to a single remote space so not sure if you will have to set-up 2 jobs or be able to copy to 2 locations with one job.

 

EDIT: Took a look, it is a VMWARE Backup Copy. The main backup job is selected as the object to process, it is a direct job and the Schedule is Any Time.

Hmm, OK. I've just had a look and the scheduling looks more promising! Would that be a solution for disaster recovery?

 

How many copies of the VM do you keep? Say Server1 performed a VM copy job to NAS1 yesterday. How do you manage the Server1 VM that copied to NAS1 yesterday? Do you keep that copy and it creates a new one? Does it overwrite the old Server1 VM file? Do you manually delete it? Do you delete it via script?

[PotNoodleTech]

Thanks for all of the replies. I have decided to scrap the fire safe idea and gone for another two NAS boxes in two other separate buildings. However I'd wondering what the best way to transfer our VEEAM backups to these NAS boxes is. So far I can only see two options:

 

1) Use VEEAM's Backup Copy job

2) Use Robocopy

 

I'd prefer to keep it all within VEEAM if I can, however VEEAM's scheduling options for Backup Copy jobs appears to be limited. For our normal backup jobs we have one job per server, so 19 jobs in total, with the first server scheduled to backup at 9pm, and the rest scheduled to start once the one before has completed. I'd have liked to replicate this within the backup copy jobs, for simplicity sake. However when a backup copy job is created, the only scheduling option is to allocate a time slot, so it is not as comprehensive scheduling backup jobs. I could create a job containing all the backup jobs, but I was hoping to avoid that if possible.

 

I can create a Robocopy job, but I'd like to have email notifications on failure, like VEEAM sends.

 

Anybody have any ideas?

 

Why would you even mention the word robocopy? Veeam backup copy job does it all plus you get total control of what kind of data retention (grandfather, father, son, etc) you have on the job in order to maximise you're use of space on the backup copy target.