Backup strategies - please discuss

[SteveLaw]

Hiya,

 

I started as Network Manager in my school in September. It's been a busy month (to say the least) and I'm only just getting around to actually looking at the system I've inherited (the previous technician - no Network Manager before - is still here so it hasn't been totally neglected...).

 

Anyway, as I discovered almost by accident at 3.30 yesterday! backup is of primary concern. There really isn't any to speak of. Over the summer the LA have set up and installed a remote backup service (Iron Mountain? - I need to arrange to find out about this system) but this is backing up a bare minimum of data from the main server. I also just found out that the majority of students' work is not even on this server, but still on the old server and consequently is not even backed up! (Naturally I copied them over manually and they are backing up as we speak, but this is only a knee-jerk temporary solution.)

 

Each server is RAID 5 (or so I'm told, I need to check for myself) so that's something at least.

 

Anyway, suffice it to say it's a bit of a mess. I need to draw up a few choices of backup strategies for when I pass on my discovery on Monday. Oh, also, the school is very short of funds so cost-effectiveness is something of a priority.

 

It would really help me if you could share some thoughts/ideas/experiences of different backup strategies. By this I include both physical media used (I'm guessing primarily tape/NAS) and type of backup method (daily full, full+incremental, full+differential, or even something like full+incremental for most but daily full for users data, etc.).

 

This is a quick rough (rounded up) snap shot of what we have on the two servers.

 

(Space is used / total available)

 

Main Server:

C: 20 GB / 250 GB

D: 180 GB / 700 GB

E: 300 GB / 700 GB

 

Old Server: (storage will not likely grow too much beyond this as I will be migrating what I can to the main server ASAP - half term hopefully)

C: 6 GB

D: 6 GB

E: 65 GB

F: 130 GB

 

I'm currently thinking of some kind of NAS box so that both can backup to it, but is this the best solution? Would it be better to have a tape drive on the main server and just backup the old server to the main to be included on the tape?

 

Etc.

 

Bit of a nightmare really, any help would be very much appreciated.

 

Cheers.

[CyberNerd]

I built a NAS from old components and added some new 1TB disks for backup. the backup server periodically gets backed up to 400GB tapes.

 

All the windows servers have cwrsync, and the *nix servers have rsync

The backup application rsback runs on the backup server overnight.

 

This method allows incremental and full backups. The use of hardlinks means that disk space is saved (30 day incremental stored one copy of files that have not changed, also each individual daily change). rsync only copies files that have changed, so backups can run very fast.

total software cost = £0

total hardware = disks, old computer, tape drive

Edited October 4, 2008 by CyberNerd

[SteveLaw]

Both servers are Windows btw. Thanks for the info though - I knew about rsync but hadn't thought of looking for a Windows port of it. The link to cwrsync didn't work, but Google does

 

(Could you explain what you mean by using Hardlinks - and how you use them - or is that something rsback handles?)

Edited October 4, 2008 by SteveLaw

[CyberNerd]

The link to cwrsync didn't work, but Google does

sorry, i monkeyed up the link in my post.

[CyberNerd]

 

(Could you explain what you mean by using Hardlinks - and how you use them - or is that something rsback handles?)

 

rsback handles the hardlinks, but the basic principal is that when a file is created the filename is simply a link to the inode where the real file is held. Several filenames can link to the real file, so for example: if 3 hardlink filenames link to one 2mb file, it would appear that there are 3 files totaling 6mb, but the total amount of diskspace used is only 2mb. If one of the files is deleted the inode (real file) remains. the real file is only deleted if all the hardlink filenames are deleted.

theres quite a good explanation here:

Easy Automated Snapshot-Style Backups with Rsync

[SteveLaw]

While I'm definitely going to think about/look into and probably include in my "report" something like CyberNerd is using, I'm a little worried about the time scale of setting it up and getting it working (among all my other duties) and that I don't think we have any old kit that could conceivably be used for this so I think off-the-shelf may be the option to go for.

 

As I'm still leaning in the NAS direction, let me narrow the subject a little to help the discussion.

 

Would I be right in thinking that the off-site element is the only real disadvantage of NAS over tape? But it has better scalability (can add bigger disks to the same box, but to scale up tape you have to replace the whole thing)? What about reliability? Tapes also add in a running cost - replacing the tapes every year (or X years?).

 

If I can tailor the online backup to backup critical data (would user documents/system state cover that?) off-site backup is not going to be a major issue, right? Is that just a disaster waiting to happen?

 

Also, with NAS backup, what backup software do people use? Would NT backup handle this or would it be better with third party?

[maniac]

Before going for a Disc to Disc option, it may be worth checking with your school insurers, as ours require a removeable backup to be taken at least once a week, and taken to an approved secure place off the school premises. Therefore we have stuck with Tape backups which are reliable if you set them up properly and clean the tape drive regulalry.

 

Each weekly backup is to be kept for a minimum of 1 month giving a minimum of 4 offsite backups at any one time. If we don't do that, and we lose data as a result of a fire or other mis-hap, then the school won't be covered for any financial losses or other issues that may result.

 

Mike.

Edited October 4, 2008 by maniac

[torledo]

There's nothing to stop you from backing up the NAS filer to tape, and have tape or other removable media in your backup workflow for offsite bakup.

 

d2d2t

[SteveLaw]

I would never have even thought about the insurance angle, thanks. Although there is an off-site backup technically (the online one). Definately worth asking though, thanks again.

[SteveLaw]

There's nothing to stop you from backing up the NAS filer to tape, and have tape or other removable media in your backup workflow for offsite bakup.

 

d2d2t

 

Ideally yes of course, but financially I'm pretty sure it's going to be one or the other.

[Midget]

what about 8 750gb HDDs?

 

label them Mon, Tue, Wed, Thu, Fri1, Fri2, Fri3, Fri4

 

get a usb SATA hotswap bay.

 

keep them in a fireproof data safe (different from a normal fire proof safe)

 

and use NTBackup (if you dont have any other backup software) to back it all up to a different HDd each day.

[dhicks]

Ideally yes of course, but financially I'm pretty sure it's going to be one or the other.

 

Could combine the two - have your NAS consist of a case with two 1TB drives in, accessible from the front in removable caddies, and set up in a RAID-1 array using a decent hardware-based RAID controller. Then have all your computers backup to that computer and when you want an off-site backup simply whip one of the caddied harddrives out, replace with a spare, and take it home.

 

Regarding software, I simply wrote my own script to send updated files from a computer (Linux or Windows) to a server and store them, using the hard links method outlined above.

 

--

David Hicks

[Guest Guest]

Over the summer the LA have set up and installed a remote backup service (Iron Mountain? - I need to arrange to find out about this system) but this is backing up a bare minimum of data from the main server.

 

we're on the same system. Ours was set up but is not working. My boss has been in contact with the LEA so i dont know what is happening personally but something to check.

 

Either way they will only backup user files and your DC. They havnt got the capacity to backup app servers yet. Iron mountain cannot backup a DC directly so you must use NTbackup to backup to a location which will then be backed up by Ironmountain.

[SteveLaw]

Thanks for a bit more info Sparky, I was dubious about System State (which is what I presume you mean?).

 

Re: removing the drives, I did think about that to be honest although something doesn't feel right about it (but I can't put my finger on it).

 

What about this one?

Edited October 4, 2008 by SteveLaw

[Guest Guest]

Thanks for a bit more info Sparky, I was dubious about System State (which is what I presume you mean?).

 

Yes thats what i mean. Couldnt think of the term at the time

[Midget]

Re: removing the drives, I did think about that to be honest although something doesn't feel right about it (but I can't put my finger on it).

What about this one?

 

cheaper than tape, and fine as long as you don't drop them, that bay will work fine i used to use one before for recovering data from sata drives.

 

the problem with NAS is that you are creating another point of failure, at least with lots of drives you increase your chances and doesn't require much money.

[SteveLaw]

cheaper than tape, and fine as long as you don't drop them, that bay will work fine i used to use one before for recovering data from sata drives.

 

the problem with NAS is that you are creating another point of failure, at least with lots of drives you increase your chances and doesn't require much money.

 

You're selling me...

[CyberNerd]

the problem with NAS is that you are creating another point of failure, at least with lots of drives you increase your chances and doesn't require much money.

 

You have the same problem with tapes failing, esp when using several tapes and an encrypted tape archive (off site backup should be encrypted, as it would be a problem if someone lost your MIS data.) I reckon It's also easier to tell if a disk drive has failed.

 

We currently backup with both NAS and then copy that to tape - but rsync to an offsite location such as amazon S3 is an attractive alternative

Amazon Simple Storage Service (Amazon S3)

[Midget]

I wasn't saying to use tape, but HDDs which are cheap and easy to check for problems and last longer than tapes.. I'm just coming up with cheap solutions for him to implement because something is better than nothing.

 

Off site backup is always a good idea, but unless you want to host it yourself (not reccomended due to legal reasons) you will be looking at a very big bill.

[CyberNerd]

I wasn't saying to use tape, but HDDs which are cheap and easy to check for problems and last longer than tapes.. I'm just coming up with cheap solutions for him to implement because something is better than nothing.

I wasn't disagreeing with you, just pointing out that some of the problems with hard disks apply equally to tape vice versa.

[SteveLaw]

because something is better than nothing.

 

Which is the key thing here, but of course I want the best something I can wangle (but without compromising purchases later in the year).

 

I want to present a range of options, starting, probably, with disk-to-disk-to-tape in some form and working down through NAS only, tape only and now also including removable disk (thanks) probably as a preference over tape (I just don't like tape to be honest).

 

I'm going to try to show the advantages and disadvantages of each choice which is where the discussion is useful.

Edited October 5, 2008 by SteveLaw

[Midget]

I wasn't disagreeing with you, just pointing out that some of the problems with hard disks apply equally to tape vice versa.

sorry, obviously missread.

 

With using 750 HDDs, you can then chuck them into a NAS and have a 5TB SATA RAID5 array if you want to change over to that.

 

Just ask yourself this question, what happens if the school burns down?

 

At least with HDDs you can take them home each day

 

but I wouldn't bother with tapes at all.

[torledo]

sorry, obviously missread.

 

With using 750 HDDs, you can then chuck them into a NAS and have a 5TB SATA RAID5 array if you want to change over to that.

 

Just ask yourself this question, what happens if the school burns down?

 

At least with HDDs you can take them home each day

 

but I wouldn't bother with tapes at all.

 

and you can't take tapes home each day ?

 

I think most of the objection to tape is based on cost and reliability.......both of those are pretty outdated arguments now that LTO-2 for external drives/loaders is the norm. LTO-2 tapes are cheap, capacity is huge and performance is blinding. A few years ago SDLT's and robotic arm tape changer we used to have all sorts of problems with but 8-disk autochangers and LTO-2 are a vast improvement and a lot cheaper to implement.

 

IF your building an openfiler why not look at incorporating an internal tape drive occuping a 5.25" bay.....and go for a cheaper drive using DDS tape.

[SteveLaw]

and you can't take tapes home each day ?

 

I think most of the objection to tape is based on cost and reliability.......both of those are pretty outdated arguments now that LTO-2 for external drives/loaders is the norm. LTO-2 tapes are cheap, capacity is huge and performance is blinding. A few years ago SDLT's and robotic arm tape changer we used to have all sorts of problems with but 8-disk autochangers and LTO-2 are a vast improvement and a lot cheaper to implement.

 

IF your building an openfiler why not look at incorporating an internal tape drive occuping a 5.25" bay.....and go for a cheaper drive using DDS tape.

 

The biggest problem I have with tape is the scalability. If you run out of space on your tapes you can't just buy a set of bigger tapes, you have to buy a new drive and a new set of tapes.

 

We'd need at least LTO-3, 400 GB native to "800" GB compressed - in practice you're probably looking at around 600 GB compressed. That's barely adequate for our needs now and I'd be loathe to pay out around £1,000 - 1,200 for maybe a years worth of backups before it needs replacing. An LTO-4 is going to be closer to £2,000. Plus £400-£500 to replace the tapes each year or so. (I'm not sure what point a DDS would be to be honest?)

 

I know it's not something you should put a price on, but if I have to struggle with removable SATA drives in order to be able to replace a couple of Pentium 3 workstations when they die later in the year, then I think I probably will. I could do with a workstation/laptop for myself but that option is going the journey this year regardless.

 

But again, it's all just about options. Ultimately I'm hoping to not have to make the final decision but only to inform those who do...

Edited October 5, 2008 by SteveLaw

[torledo]

The biggest problem I have with tape is the scalability. If you run out of space on your tapes you can't just buy a set of bigger tapes, you have to buy a new drive and a new set of tapes.

 

We'd need at least LTO-3, 400 GB native to "800" GB compressed - in practice you're probably looking at around 600 GB compressed. That's barely adequate for our needs now and I'd be loathe to pay out around £1,000 - 1,200 for maybe a years worth of backups before it needs replacing. An LTO-4 is going to be closer to £2,000. Plus £400-£500 to replace the tapes each year or so. (I'm not sure what point a DDS would be to be honest?)

 

I know it's not something you should put a price on, but if I have to struggle with removable SATA drives in order to be able to replace a couple of Pentium 3 workstations when they die later in the year, then I think I probably will. I could do with a workstation/laptop for myself but that option is going the journey this year regardless.

 

But again, it's all just about options. Ultimately I'm hoping to not have to make the final decision but only to inform those who do...

 

Or just buy a tape autoloader for say 1500 pounds. For even better scalability you can get a SAS autoloader that can scale to two drives and about 16 slots. Now i know that's a lot of money for offloading data from a NAS device, so i understand why it's not that attractive. But when you investigate properly from a price, performance and scalability perspective tape products for SMB's these days measure up quite well. They really do. The reason big storage environments don't like tape is because they have to handle literally dozens of tapes on a daily basis......and their backup windows are becoming shorter and shorter, so they do use disk and offsite vaulting to not have to deal with tape turnaround. But they've still got tape in there and a lot of it.