What RAID setup do you use in your school?

[PotNoodleTech]

So can proxmox be installed on a bog standard poweredge 740 or the like?

[mavhc]

Yeah, it's just debian. https://www.proxmox.com/en/products/proxmox-virtual-environment/requirements

 

I bought SSDs, booted off sdcard/usb, installed on ssds, mounted the old ntfs hyperv hw raid hdd drives, copied the old VMs to ssd, converted file formats to kvm's format and booted up again, took about a day for each server.

 

Didn't need to buy support, but you can https://www.proxmox.com/en/services/support-services/support

[dmj]

43 minutes ago, Rob_D said:

The same could be said for server virtualization, that took us 4-5 months. You're still talking about formatting down and rebuilding the hosts, then migrating all the VMS across.

I'm not saying it's not doable, or event that technically complex, but it's still a pretty big job.  

 

It all depends on use case, but in the context of the OP's question about implementing RAID and backups its the solution I would go with. 

[PotNoodleTech]

Thanks I will give it a go on a spare server. 

[BKGarry]

On 31/03/2025 at 09:14, supportman said:

Never used raid after having a RAID card failure early in my career. 

 

Old RAID Card failures were the worse, modern RAID cards actually keep a bit of info on a disk, so as long as the disks are in the same order and it is the same make/model card it will read the RAID config from the disks.

 

Myself, I don't actually have RAID RAID, except for the OS Disks, which are mirrored, I have the rest of the disks on my two hosts as HBA, and have it configured as a hyper converged cluster

[kevin_lane]

We have a 5 node hyper v S2D cluster in a mirror setup with no hardware raid feels risky seems to be ok for now slowly moving vms over but I also have ssd disks on standby should something happen down the line 

 

The OS are hardware raid obviously 

[localzuk]

On 02/04/2025 at 08:47, mavhc said:

That's the point though, ZFS isn't new, it's not bleeding edge.

In the context of a Windows-centric network? Yes it is. OpenZFS only released their first RC for Windows in 2023. It is absolutely bleeding edge.

[dmj]

4 minutes ago, localzuk said:

In the context of a Windows-centric network? Yes it is. OpenZFS only released their first RC for Windows in 2023. It is absolutely bleeding edge.

I ran part of a windows-centric university, 20,000 users (mostly windows, and mac). 

All our servers ran from ZFS, all our file systems ran from ZFS. It wasn't bleeding edge at all.

[mavhc]

No one suggested using ZFS on Windows though, Windows is just a VM on top of ZFS

[localzuk]

On 24/04/2025 at 09:27, dmj said:

I ran part of a windows-centric university, 20,000 users (mostly windows, and mac). 

All our servers ran from ZFS, all our file systems ran from ZFS. It wasn't bleeding edge at all.

 

53 minutes ago, mavhc said:

No one suggested using ZFS on Windows though, Windows is just a VM on top of ZFS

So, both of you seem to live in a world where all techs have the time and inclination to introduce things that are not Windows native. E.g. introducing Linux/BSD or whatever servers, introducing new file systems, introducing tech that isn't directly supported by support contracts (e.g. if you buy a SAN).

 

So, either ZFS is bleeding edge, in Windows, or it is not in Windows and is therefore not Windows-centric (and therefore the staff running it do not have the knowledge and experience to run it).

[psydii]

On 24/04/2025 at 09:27, dmj said:

I ran part of a...[organisation with]...20,000 users...

Scale! Most here have 10 times fewer users.

 

If one has 20,000 users-colleagues one is  likely have a budget large enough to have a couple of people on staff who are (somewhat) dedicated to the server/storage infrastructure (or already be fully cloud/SAAS) . The rest of us need point-and-click, setup-once-and-never-think-about-it-again, where SLT can hire in any old tech from ESP  who can rock up see the blinking red lights and pick up the phone to HP / Dell / Lenovo and get immediate support.

 

I didn't roll our own storage/virtualisation platform with proxymox/zfs for the same reason I didn't with hyper-v/storage spaces direct.  *I know i COULD have* but I don't trust the organisation to be able to hire in alternatives with the right skills should things go sideways (sudden death due to an unfortunate encounter with the Clapham Ominbus, annual leave etc). Instead we bought in basically a standard solution from a vendor that looked exactly like the standard solution from all the other vendors. 

 

There is some irony of course, now Broadcom have thrown the SME market under the aforementioned Clapham Omnibus. Perhaps next time I consider storage/servers I'll end up receiving quotes for  a load of proxmox/zfs based solutions

 

[dmj]

1 hour ago, localzuk said:

 

So, both of you seem to live in a world where all techs have the time and inclination to introduce things that are not Windows native. E.g. introducing Linux/BSD or whatever servers, introducing new file systems, introducing tech that isn't directly supported by support contracts (e.g. if you buy a SAN).

 

So, either ZFS is bleeding edge, in Windows, or it is not in Windows and is therefore not Windows-centric (and therefore the staff running it do not have the knowledge and experience to run it).

 

It's definitely supported, just buy a QNAP or a support contract if you want that and don't have the time or inclination to learn a better technology. Or carry on like you always have and not change anything. 

[mavhc]

Yeah, me with very few users and him with very many users manage, it's only the people with exactly the wrong number of users who seem have problems.

 

If you don't know Linux in 2025 you're outsourcing your job, most popular OS on 2 planets

[psydii]

18 minutes ago, mavhc said:

"you're outsourcing your job"

Yup. If things go sideways, I *want* it to be someone else's job.

 

I also want to be able to walk away at some point and have reasonable confidence that SLT can hire a suitable replacement. Maybe it is indeed because I sit in an org that is not small, but not huge: I have too many things under my remit to expect "virtualisation and storage guru" to *necessarily* be on the CV of otherwise capable candidates. Indeed many things I am responsible for were designed, implemented and backed by 3rd party support to avoid having an impossible list of "must-haves" on the JD.

[dmj]

On 25/04/2025 at 15:59, psydii said:

Yup. If things go sideways, I *want* it to be someone else's job.

 

I also want to be able to walk away at some point and have reasonable confidence that SLT can hire a suitable replacement. Maybe it is indeed because I sit in an org that is not small, but not huge: I have too many things under my remit to expect "virtualisation and storage guru" to *necessarily* be on the CV of otherwise capable candidates. Indeed many things I am responsible for were designed, implemented and backed by 3rd party support to avoid having an impossible list of "must-haves" on the JD.

Just because a technology is different doesn't make it harder or any less well supported.

We're not talking about some obscure edge case here, it pretty standard and well documented stuff that's been around for decades. Anybody who can read the docs for a RAID controller or veeam can do so for ZFS. 90% of AWS/GCP workloads are linux and 60% on Azure, the idea that this sort of thing is unsupported, not backed by third parties and there's too few skilled people is just absurd. 

[psydii]

On 27/04/2025 at 09:15, dmj said:

Anybody who can read the docs for  [any technology at all]

This can actually be quite a high bar given the sort of money many in this sector thinks is appropriate to spend on IT Support.

 

Of course if I buy a NAS/SAN and its underlying tech is ZFS, that's great, the vendor will support it and my school.  But roll-your own storage platform is perhaps something that school tech's shouldn't be doing.

 

At this point though, short of support staff suppliers like ESP, or school IT solution providers like XMA weighing in with how many of their people have the level of ZFS (or Storage Spaces Direct) knowledge that could replace what you get with 24/7/15min HP SAN/Server support, I think we are at an impasse.

 

 

[Hoyt]

It's interesting for me to see the city folks talk about staffing qualifications.  As some have said the knowledge required for a lot of this stuff being discussed is just not available in certain areas regardless of salary available.  Where I work I.T. staff start out at about $35,000 a year, which I guess would be about 26,000 pounds and it's a 12 month position.  The type of knowledge discussed in this thread certainly will not be available for that figure.

 

The basic RAID 5 and lately RAID 6 configs have not lost me any data in the twelve plus years I have been here using the basic Windows Server OS.  I have also recovered a lot of "lost" user data using volume shadow copies without having to resort to our backups.

 

For what I am getting paid I am happy with this arrangement.  I imagine that if I can pull it off other low level techs could as well without leaving a confusing mess for someone that has to follow along behind them.

[Koldov]

I guess it's a question of whether you believe you are doing the best for your school, yourself, or your successor... is a simple set-up bad if it does the job? Does using the same infrastructure solutions that enterprise use for tens of thousands of worldwide users for c.1000 users on one site really make things better? Or does it just make you feel like a cool, bad-ass sysadmin?  iirc someone was saying that they'd  had a set-up implemented that enabled 15 minute snap-shot backups... or that they could be backup and running in half an hour after some sort of server failure... can this not be done on a 'traditional' server set-up with RAID then...?

 

It's an interesting subject, maybe worth its own thread as it's a bit more involved and off-topic than the discussion on the benefits/downfalls of RAID 5, 6 or 10, but it would be one of my worst nightmares to lose my job here and have to go into another school where the tech has had free rein with no oversight for years (I mean I know we all do to some extent because we are the only ones who know what we're doing)...

 

It would be daunting to start a job in a place where everything is running on linux servers, bespoke software and open source projects, or cheaper unsupported 'alternative' versions. All of it requiring downloading and implementing various packages from Github or the like, a healthy dose of coding and a fair amount of Enterprise grade infrastructure knowledge... I mean I guess there would be copious amounts of documentation to go through before you'd dare push a button... and we all know there'd be no time for that as you'd have people banging on your door from day 1...

 

I mean we've probably all got a few workarounds that we've had to use to get the software used by the Science Dept. running... but I don't know if I should actually be worried that I can almost guarantee any one of you could walk in here and have it all under your belt in an hour or so... I guess that makes me very easily replaceable, but luckily even this simple set-up is rocket science to the SLT! 

 

EDIT: By the way for the OP - RAID 10

Edited April 28, 2025 by Koldov

[Hoyt]

Koldov you just brought something to mind.  I don't really want to be the only guy who knows how things work around here, because if I was, my workload would grow tenfold.

 

Our district is split into four quadrants with one tech in each one looking after between five and seven schools each.  Being the oldest and most experienced of the four I always tell them that there is nothing I know (privileged info aside) that I do not want them to know.  Unfortunately it seems these days that curiosity may have died with the cat.  I get very few questions until something breaks and I get told by the Boss to look into it.

 

Anyhow I have enjoyed this thread.  I don't have any exposure to most of the topics being discussed so I do appreciate the info.

[PotNoodleTech]

I think the key thing to remember is that you are taking about replacing a bit of technology that's built into almost every server you can buy that isn't garbage, it actually works really well and really reliabily and has done for many decades.  It's not a black and white decision.

 

Having not lost a single RAID array in 24 years I have to say I would need some convincing.  But there are very convincing argments above.  If I was starting from scratch now with my infrastructure I would look, and try ZFS.

 

It reminds me of a saying my good old Head of DT used to say back in the day.  he had a saying for every situation,

 

KISS.  Keep It Simple, Stupid - I agree with this as the more complex you make something the more unreliability and difficulty you introduce.  This is why I have always disliked SANS etc as they dont really bring much to the table other than significant extra complexity and cost.  I did view ZFS as similar but my mind is changed now.

Edited April 29, 2025 by PotNoodleTech

[mavhc]

20 hours ago, Koldov said:

iirc someone was saying that they'd  had a set-up implemented that enabled 15 minute snap-shot backups... or that they could be backup and running in half an hour after some sort of server failure... can this not be done on a 'traditional' server set-up with RAID then...?

 

First problem with HW RAID is now you have a single point of failure. Second problem is you have a proprietary disk format, will your disks on one RAID card work on another?. 3rd is RAID is super dumb, when restoring a disk it'll copy every sector, not just the used ones.

 

How do you restore your backup? Does it image the whole disk? The filesystem? Each file? for ZFS the answer is yes to all 3 at the same time. How are diffs to your backup handled? Do you have to restore the full backup, and then each diff?

 

20 hours ago, Koldov said:

It would be daunting to start a job in a place where everything is running on linux servers, bespoke software and open source projects, or cheaper unsupported 'alternative' versions. All of it requiring downloading and implementing various packages from Github or the like, a healthy dose of coding and a fair amount of Enterprise grade infrastructure knowledge... I mean I guess there would be copious amounts of documentation to go through before you'd dare push a button... and we all know there'd be no time for that as you'd have people banging on your door from day 1...

 

Take a random thing you want some software to do, where there are many many choices, let's say 2 are open source, and there's 10 closed source options, and you move to a job that uses one of them, what's the chance you've used it before?

 

Hypervisors: Hyper-V, KVM, ESX, Xen. 75% of those were Linux based, although ESX isn't now.

 

Linux is the most popular server OS, so you should know that if you work with servers. 

 

Deploying software is mostly just copying and pasting docker-compose files nowadays, install a linux vm, docker, portainer, and then it's all a web UI, easy peasy

[dmj]

12 minutes ago, mavhc said:

 

First problem with HW RAID is now you have a single point of failure. Second problem is you have a proprietary disk format, will your disks on one RAID card work on another?. 3rd is RAID is super dumb, when restoring a disk it'll copy every sector, not just the used ones.

 

How do you restore your backup? Does it image the whole disk? The filesystem? Each file? for ZFS the answer is yes to all 3 at the same time. How are diffs to your backup handled? Do you have to restore the full backup, and then each diff?

 

 

Yeah, why learn and pay for support on 4 different things: RAID, Backup, Snapshot, Filesystem when you can just learn a modern filesystem !

 

[dmj]

4 hours ago, PotNoodleTech said:

KISS.  Keep It Simple, Stupid - I agree with this as the more complex you make something the more unreliability and difficulty you introduce.

100% that's why I use ZFS - it does all the things I need without having to string together disparate systems

[FN-GM]

On 31/03/2025 at 18:44, supportman said:

Never used raid after having a RAID card failure early in my career. 

 

Basically we use fast virtualised machines on SSD drives with daily backups to veeam.

 

I don't see the issue. We have had a RAID card fail, swapped it out and we are up and running again.

[NegativeKillDeath]

FN-GM likes raid so much they post in RAID 1