Single disk server?

[mavhc]

Really you should be using ZFS, only needs to rebuild the parts of the drives with files on. Plus checksums on data, snapshots, and virtual file systems.

[JackCPickup]

Really you should be using ZFS, only needs to rebuild the parts of the drives with files on. Plus checksums on data, snapshots, and virtual file systems.

 

ZFS is just the file system and LVM. What ZFS RAID level are you recommending?

 

ZFS RAID levels are basically the same as regular RAID levels with added checksumming, which actually causes a performance loss at some levels. ZFS doesn't make sense for all platforms.

[mavhc]

ZFS is just the file system and LVM. What ZFS RAID level are you recommending?

 

ZFS RAID levels are basically the same as regular RAID levels with added checksumming, which actually causes a performance loss at some levels. ZFS doesn't make sense for all platforms.

 

Triple parity of course! Or D-RAID if you want speed over space usage.

 

I had a look at running SmartOS, but its migration of VMs is still experimental, they're more into stateless stuff.

[JackCPickup]

Triple parity of course! Or D-RAID if you want speed over space usage.

 

I had a look at running SmartOS, but its migration of VMs is still experimental, they're more into stateless stuff.

 

Then I'm not sure I understand your comment of what they should "really" be doing.

 

Are you saying ZFS RAID is the only worthwhile production use RAID? That it does something incredible that other software RAID doesn't? Because those statements aren't true.

 

Especially for your typical school IT experience/knowledge pool, and infrastructure needs it is the least appropriate recommendation.

 

Though schools users typically outnumber SMBs, their LAN infrastructure needs are pretty equivalent if not heavier on the side of SMB depending on the industry, so I always think a good direction to go is look at SMB best practices and use your wits to figure what best meets your needs there.

Edited November 10, 2017 by JackCPickup

[PotNoodleTech]

Great thread. There's so many ways of doing storage.

 

One question I would ask is: Have you ever had a single disk fail? If so - then would you want a server to be running on a single disk?

 

I have had single disks failed of all different types from 7200 sata to 15k RPM enterprise to SSD. All can and will fail at some point. I wouldn't use a single disk in a server.

 

For me I went with SSD drives in Raid 5. Why raid 5 you ask? Isn't raid 5 a dirty word? Because it's the ideal raid group for SSDs.

1. you get resiliency, if one drive fails the array stays up.

2. You inherit raid 5's drastic increases in read speeds, and most usage in schools is read

3. Unlike other raid types, raid 5 it reduces the number of writes on each drive, (writes are splitequally over N+1 drives) therefore theoretically increasing each SSD's lifespan.

4. If you lose a drive, SSD's are so fast the "risk window" of losing a second drive at the same time is minimised, unlike sata drives which can have a risk window of 24-48 hours.

5. You maximise storage space vs theoretical risk of array failure. Only losing 1 drive's worth of data is a big thing with expensive SSD's compared to Raid 6 or Raid 1/01/10.

 

That's my 2p!

 

It all comes down to Risk vs Cost and for me the above ticks all the boxes. TOUCCH WOOD I've never lost a Raid 5 array in nearly 20 years of using Raid5 arrays. TOUCH WOOD.

 

*awaits the inevitable raid failure now i've said the above*

[JackCPickup]

Great thread. There's so many ways of doing storage.

 

One question I would ask is: Have you ever had a single disk fail? If so - then would you want a server to be running on a single disk?

 

I have had single disks failed of all different types from 7200 sata to 15k RPM enterprise to SSD. All can and will fail at some point. I wouldn't use a single disk in a server.

 

For me I went with SSD drives in Raid 5. Why raid 5 you ask? Isn't raid 5 a dirty word? Because it's the ideal raid group for SSDs.

1. you get resiliency, if one drive fails the array stays up.

2. You inherit raid 5's drastic increases in read speeds, and most usage in schools is read

3. Unlike other raid types, raid 5 it reduces the number of writes on each drive, (writes are splitequally over N+1 drives) therefore theoretically increasing each SSD's lifespan.

4. If you lose a drive, SSD's are so fast the "risk window" of losing a second drive at the same time is minimised, unlike sata drives which can have a risk window of 24-48 hours.

5. You maximise storage space vs theoretical risk of array failure. Only losing 1 drive's worth of data is a big thing with expensive SSD's compared to Raid 6 or Raid 1/01/10.

 

That's my 2p!

 

It all comes down to Risk vs Cost and for me the above ticks all the boxes. TOUCCH WOOD I've never lost a Raid 5 array in nearly 20 years of using Raid5 arrays. TOUCH WOOD.

 

*awaits the inevitable raid failure now i've said the above*

 

Completely agree with the recommendation and statements. Just one thing I'd like to clarify, while there is definitely a speed increase because of the amount of drives, there is still a large penalty because of writing and reading parity. Of course though you're going to be fine with this as overall it's faster than a single SSD and the storage you get is great!

[mavhc]

Then I'm not sure I understand your comment of what they should "really" be doing.

 

Are you saying ZFS RAID is the only worthwhile production use RAID? That it does something incredible that other software RAID doesn't? Because those statements aren't true.

 

Especially for your typical school IT experience/knowledge pool, and infrastructure needs it is the least appropriate recommendation.

 

Though schools users typically outnumber SMBs, their LAN infrastructure needs are pretty equivalent if not heavier on the side of SMB depending on the industry, so I always think a good direction to go is look at SMB best practices and use your wits to figure what best meets your needs there.

 

So, you're either using HyperV with hardware raid, or dynamic disks for software raid (probably mirrored), or storage spaces/ReFS

Or you're using VMWare, which doesn't do software raid, so you might be running diskless with a SAN (which is hopefully doing software raid with data checksumming)

Or KVM/Proxmox/Xen/SmartOS/Nexenta which can all use ZFS afaik.

 

You may be doing Direct IO to map physical storage drives into a VM, in which case you have more options, zfs in the VM for example.

 

ReFS will checksum your data if you enable it, but it's still in massive development. Should be ready in a few years.

 

Checksumming data is super useful for RAID 1, as you only read from 1 drive at a time usually, if one is corrupt you can't tell which.

[JackCPickup]

So, you're either using HyperV with hardware raid, or dynamic disks for software raid (probably mirrored), or storage spaces/ReFS

Or you're using VMWare, which doesn't do software raid, so you might be running diskless with a SAN (which is hopefully doing software raid with data checksumming)

Or KVM/Proxmox/Xen/SmartOS/Nexenta which can all use ZFS afaik.

 

You may be doing Direct IO to map physical storage drives into a VM, in which case you have more options, zfs in the VM for example.

 

ReFS will checksum your data if you enable it, but it's still in massive development. Should be ready in a few years.

 

Checksumming data is super useful for RAID 1, as you only read from 1 drive at a time usually, if one is corrupt you can't tell which.

 

 

All very interesting facts but I'm just not sure how that results in a "really you should be doing..." when applied to the context of school infrastructure. It's sometimes too easy to over-engineer.

 

The checksumming feature of ZFS, and others of ZFS, make it a great file system. But they should be absolutely minute details in choosing your platform, filesystem should not dictate your platform choice. If you're having so much bit rot with best practice RAID levels, practices, and hardware then you've got other issues to worry about.

 

Platform before fs and questioning your "really" comment are my main points. It gives the impression of there being only one right way which can play with people's naiveties and send them in the wrong direction.

[zag]

Wow you guys like to complicate things!

 

I'll go back to what I said earlier. A single disk is absolutely fine these days with a good virtual infrastructure with backup.

[JackCPickup]

Wow you guys like to complicate things!

 

I'll go back to what I said earlier. A single disk is absolutely fine these days with a good virtual infrastructure with backup.

 

RAID usage for the context here is not complicated at all and should be seen as an absolute necessity, you've got to be sinking extra unnecessary costs elsewhere to make up for the lack of redundancy, or you've just decided you aren't bothered about redundancy? It is the most basic and crucial level of redundancy in your infrastructure.

 

A single disk is absolutely not fine "these days" and you should be careful about broadcasting that on a professional IT forum, where lots of people new to IT come to learn about the industry and best practices.

Edited November 10, 2017 by JackCPickup

[ThatBoringBloke]

Wow you guys like to complicate things!

 

I'll go back to what I said earlier. A single disk is absolutely fine these days with a good virtual infrastructure with backup.

 

Would you run a Hyper-V host server hosting at least 4 VM's using local storage on a single disk server?

 

What would you define as a "Good virtual infrastructure"?

[3s-gtech]

Simplifying a bit:

 

A single disk is absolutely fine these days with a good virtual infrastructure with backup.

 

What may be fine for your use case may not be recommendable as a general case. I think you need to start it with "I believe..." or "My findings are..." otherwise you're stating opinion as fact, and that's a mistake.

 

Basically - the consensus seems to be 'don't use a single disk'. Whether you use RAID, storage spaces, hardware or software, disks are cheap and generally they break (and sometimes, they're dead out of the box!) so buy more than one and reduce your stress levels.

[zag]

Would you run a Hyper-V host server hosting at least 4 VM's using local storage on a single disk server?

 

What would you define as a "Good virtual infrastructure"?

 

Yes I've run up to about 10 I think on local storage without issue. I've found RAM is the only real limiting factor for the number of VM's.

 

I even wrote about some of my experiences here: http://www.edugeek.net/forums/behind-red-door/131323-building-stable-ict-systems.html

 

Good virtual infrastructure (in my humble opinion) = balanced servers, instant backup, spare hardware to migrate, spare parts to fix, common hardware for hosts and solid state storage at every level.

 

All these ideas about SANS, replication, hardware level RAID would be great if I worked for a multi national company in a data center, but in reality I'm responisible for about 1,200 users and design my infrastructure to match this requirement

 

*Not saying the others are wrong of course, but my philosophy has always been to keep things simple, stable and blazingly fast.

[ThatBoringBloke]

Simplifying a bit:

 

Basically - the consensus seems to be 'don't use a single disk'. Whether you use RAID, storage spaces, hardware or software, disks are cheap and generally they break (and sometimes, they're dead out of the box!) so buy more than one and reduce your stress levels.

 

It was never a realistic option to be honest, I just had to throw it out there to gauge thoughts.

[psydii]

Risk management. What is more likely, disk failure or server failure? Does the complexity of a replicating system add risk? How much additional specialist knowledge does it take to maintain the health of a hyper-v replicates system vs a single system with hardware redundancy?

 

With replication to a “second site” you are (from some perspectives) hedging against an extremely unlikely scenario (loss of the primary site) and so this capability should not factor heavily in evaluating this option.

 

What happens if your replicas go split-brain on you?

 

Do the benefits out way the risks?

 

Quite possibly they do. It always depends.

[JackCPickup]

Yes I've run up to about 10 I think on local storage without issue. I've found RAM is the only real limiting factor for the number of VM's.

 

I even wrote about some of my experiences here: http://www.edugeek.net/forums/behind-red-door/131323-building-stable-ict-systems.html

 

Good virtual infrastructure (in my humble opinion) = balanced servers, instant backup, spare hardware to migrate, spare parts to fix, common hardware for hosts and solid state storage at every level.

 

All these ideas about SANS, replication, hardware level RAID would be great if I worked for a multi national company in a data center, but in reality I'm responisible for about 1,200 users and design my infrastructure to match this requirement

 

*Not saying the others are wrong of course, but my philosophy has always been to keep things simple, stable and blazingly fast.

 

I agree that I would absolutely not be recommending SANs or hardware RAID for any school, replication potentially. The basic use of RAID has barely any complexity and is applicable to all levels of infrastructure, nothing to do with if you're a datacentre.

 

If you're able to put that much money into your storage you should at least implement some basic redundancy between disks, you don't want to be relying on doing full restores every time you have a faulty disk. It is a bad business decision for the school if they have any concern about downtime of their systems...which I imagine they do.

[zag]

If you're able to put that much money into your storage you should at least implement some basic redundancy between disks, you don't want to be relying on doing full restores every time you have a faulty disk. It is a bad business decision for the school if they have any concern about downtime of their systems...which I imagine they do.

 

Well that's a good point but why not? A full restore of our file server SSD >> SSD takes around 55mins. Not that we've ever had a disk failure with our Intel SSD's in the last 5 years but I will entertain the possibility

 

If you look at the stats, for RAID you are going to need at least 3 disks probably. That means your storage is 3 times more likely to fail at some point.

 

Personally I would much rather just restore our file server to a dedicated host with 1 click, than start messing around with RAID in the bios or replacing hard disks.

 

The downside is, of course the potential time it takes to restore, but to be honest our staff can handle 55mins of no file server. Its very unlikely to happen with the reliability of SSD storage but if it did then its not the end of the world.

 

Just as an example, our finance server failed a few years ago and we restored the entire thing in 15mins using Veeam to a separate host. I've spent longer, trying to work out a problem just because a service has stopped or something

[JackCPickup]

Well that's a good point but why not? A full restore of our file server SSD >> SSD takes around 55mins. Not that we've ever had a disk failure with our Intel SSD's in the last 5 years but I will entertain the possibility

 

If you look at the stats, for RAID you are going to need at least 3 disks probably. That means your storage is 3 times more likely to fail at some point.

 

Personally I would much rather just restore our file server to a dedicated host with 1 click, than start messing around with RAID in the bios or replacing hard disks.

 

The downside is, of course the potential time it takes to restore, but to be honest our staff can handle 55mins of no file server. Its very unlikely to happen with the reliability of SSD storage but if it did then its not the end of the world.

 

Just as an example, our finance server failed a few years ago and we restored the entire thing in 15mins using Veeam to a separate host. I've spent longer, trying to work out a problem just because a service has stopped or something

 

Are you not running many LoB applications? What if it is a time of year when any downtime isn't ok for those above? You may get completely lucky and never have a failure for as long as you're in that job, but that isn't good risk analysis, and again it plays with people's naivety because lots of people will read that and think going forward they don't need to implement RAID without thinking about it. I think it's much better to set the standard at "You need to be implementing RAID" then as they become more experienced/knowledgeable they're in a better place to make infrastructure decisions.

[3s-gtech]

If you look at the stats, for RAID you are going to need at least 3 disks probably. That means your storage is 3 times more likely to fail at some point.

 

Woooah there. That's not how RAID works, nor Maths. You are reducing the probability of an array failing, not disk. So a single disk array vs a three disk array (where one can fail and the array continues to work) would be a probability of 1 (worst case, it definitely will fail) vs a probability of 1/3 (where one will definitely fail). It gets far more complex than that in reality but it's 'numerator over denominator', not 'numerator times denominator'!

[mavhc]

Yes I've run up to about 10 I think on local storage without issue. I've found RAM is the only real limiting factor for the number of VM's.

 

I even wrote about some of my experiences here: http://www.edugeek.net/forums/behind-red-door/131323-building-stable-ict-systems.html

 

Good virtual infrastructure (in my humble opinion) = balanced servers, instant backup, spare hardware to migrate, spare parts to fix, common hardware for hosts and solid state storage at every level.

 

All these ideas about SANS, replication, hardware level RAID would be great if I worked for a multi national company in a data center, but in reality I'm responisible for about 1,200 users and design my infrastructure to match this requirement

 

*Not saying the others are wrong of course, but my philosophy has always been to keep things simple, stable and blazingly fast.

 

I was surprised to read you had 8 physical servers, seems a lot.

 

How do you use them? How many SSDs per server? How much user storage? Are backups also SSD? Do you keep long term backups?

[DJ-1701]

Wow you guys like to complicate things!

 

I'll go back to what I said earlier. A single disk is absolutely fine these days with a good virtual infrastructure with backup.

 

Timmmmmbeeeeeeerrrrrrrrrrrrrrrrr...

[mavhc]

All very interesting facts but I'm just not sure how that results in a "really you should be doing..." when applied to the context of school infrastructure. It's sometimes too easy to over-engineer.

 

The checksumming feature of ZFS, and others of ZFS, make it a great file system. But they should be absolutely minute details in choosing your platform, filesystem should not dictate your platform choice. If you're having so much bit rot with best practice RAID levels, practices, and hardware then you've got other issues to worry about.

 

Platform before fs and questioning your "really" comment are my main points. It gives the impression of there being only one right way which can play with people's naiveties and send them in the wrong direction.

 

Ah, I wasn't taking it entirely seriously, I guess if I were I would have written "Don't forget to consider ZFS if you need more data reliability". COW snapshots of VMs is another thing to look at.

 

The platform is mostly the hypervisor, a minor part of the complexity.

[Boredguy]

SSD's can fail just as much as a mechanical HDD, and using a minimum of Raid 1 to ensure the data is available across 2 disks in case of a drive failure is the minimum I'd want on any server regardless of backup infrastructure.

[mavhc]

The 3 points for users and data are:

 

1. I can't get to my data, something's failed, how often will that happen, and how long can they wait?

2. I deleted/overwrote my data, is there an old copy?

3. My data is corrupt, arrrrgh.

 

With a possible 4th: I can't get to my data fast enough, and 5th: I can't get to my data when I'm at home/on a boat

 

So find out how long they can wait for 1. Cost up how much 2 will be. And reduce 3 by not cheaping out on hardware and having checksums, so at least you know it's corrupt. Unknown bad data is worse than known bad data.

 

At home I've stored about 40TBs of data, had about 30MBs fixed with checksums, and got a list of about 50 files that were corrupted due to 2 drives failing, over the past 8 years.

 

As always you just have to price it out, "it'll cost 1k if they can wait a day, 2k if they can only wait an hour, 100k if they can only wait a second, pick one"

[geezersoft]

If you have a server located somewhere else on the same site, this is for business continuity, not disaster recovery