New Server, Sanity Check on Spec

[craig-schools]

I'm in a large Primary School, 400 students (Foundation - Year 6) with 55 staff, I've started the groundwork for a new server in the summer. Here's the breakdown on what I'm running currently all Server 2008 R2.

 

Dell PowerEdge T610 HyperV Host ONLY

2x Xeon E5620 @2.4Ghz

16GB RAM (15GB used)

OS RAID - 2x 136GB SAS 15k (50Gb Used)

STORAGE RAID - 5x 450GB SAS 15k (including 1x HotSpare) 1.2Tb total. 1.1Tb used.

Dual LAN, Dual PSU,

5yrs old in Sept Out of Warranty.

 

HyperV VMs

AV - Kaspersky

DC1 - DNS, DHCP, AD etc. Veeam Trial backs up to local and NAS weeknights

Deployment - MDT, WSUS. Veeam Mon & Fri

Files -500Gb, holds Home Drive and Shares. Windows Server Backup to Local, Shares copied nightly to NAS, Veeam Mon & Fri

APP - runs Tucasi, Paxton and a few other bits. Veeam Mon & Fri

Print Server - Veeam Sun

SIMS Server - Scripted backup to Encrypted USB daily, Backup to NAS nightly, Veeam Mon & Fri

DC2 - Veeam Sat & Weds

Build - used for updating MDT images when I get time, maybe twice a year. Could probably use a desktop tbh.

 

Just to be clear, Veeam is running on another device. Once the above has been retired I plan on using that as the Veeam Backup Server in the other building with DC2 HyperV running.

 

New Server Spec

Server 2012

2x Xeon 2630 @ 2.4ghz

64GB RAM (was originally 32GB)

OS - 2x 160GB SSD Originally 1x SSD

STORAGE RAID - 6x 600GB SAS 15k (no hotspare) 3Tb total.

Dual LAN, Dual PSU etc.

5yr NBD Warranty.

1500VA UPS

 

I'll be purchasing End of May / Early June so that it arrives in plenty of time for the Summer break.

 

Have I over / under specced the server for 5 years service? Any suggestions that I may have overlooked?

Edited May 6, 2015 by craig-schools

[Steve21]

Personally, more RAM would be better and future proof you

 

Wouldn't trust OS on a single disk, so mirror it etc if you're keeping data on seperate RAID.

 

From your 3TB comment I'm guessing that's raid 5, would change to raid 6.

 

Also assume you mean 2012R2 rather than 2012? (Maybe me being picky but never know haha)

 

Steve

[craig-schools]

Yeh I was debating on the RAM, but I can't foresee many more VM's being required, but I take your point.

 

As for single disk, with SSD's being so reliable I wasn't sure I could get away with it.

 

Yes RAID 5, and yes 2012R2.

[Steve21]

RAM wise though you can start playing with dedup etc which eats more RAM but is shiny as shiny can be Also generally it's so cheap to add at the start it's certainly worth having it there even if it's just sitting idle on SIMs etc

 

And certainly wouldn't say even SSD is worth the risk of losing your whole HOST in 1 hit Kind of makes the rest of the redundancy pointless.

 

Steve

[craig-schools]

True. I'll make it 2x SSD's, and thats only because I know 300GB SAS would be a total waste for just the Host OS.

[dhicks]

Have I over / under specced the server for 5 years service? Any suggestions that I may have overlooked?

 

As already suggested, you might as well fit as much RAM as possible now - we found 96GB to be cheap enough, although upwards of that the price rose more sharply.

 

If this is for another Hyper-V host, is it worth setting up an SSD / SATA tiered storage array? We're going to try it out on a server here (I fitted the disks today, might get to install the OS tomorrow) with 4 SSDs and 4 larger harddives (you could go for 4TB). Other people seem to have had good results - I/O throughput near that of an SSD-only array, but with much greater capacity for your money.

[Arthur]

Build - used for updating MDT images when I get time, maybe twice a year. Could probably use a desktop tbh.

FWIW, reference images should always be created in a VM. Never on physical hardware (unless you mean a VM running on a desktop?).

[robjduk]

Far more RAM. 64 or 128GB should not break the bank.

[tmcd35]

If this is for another Hyper-V host, is it worth setting up an SSD / SATA tiered storage array? We're going to try it out on a server here (I fitted the disks today, might get to install the OS tomorrow) with 4 SSDs and 4 larger harddives (you could go for 4TB). Other people seem to have had good results - I/O throughput near that of an SSD-only array, but with much greater capacity for your money.

 

Yeah, I beginning to favour SSD/SATA tiers using Server2012 storage pools. I reckon 4x2Tb WD RE4's with a 100Gb Intel S3700 on a HBA controller in a storage pool would be cheaper than 6x600GB 15k SAS drives on a RAID controller. Given the size of the school I'd be surprised if there was a performance issue on that set up.

[craig-schools]

dhicks-

Yes it will be a direct replacement server wise so running the same amount of VM's if not a couple more as time goes on.

I've heard about the Tiered Storage that 2012 offers but admit I've not had chance to read up on it, I guess thats on my joblist for today.

 

Arthur-

The reason I said use a desktop is it would be much quicker. When I update my images the VM seems to take an absolute age to do ANYTHING, in the time it takes the VM to do anything I can image 3x laptops and do all the updates etc. I assume its thrashing the drives pulling WSUS from a VM and then installing them on another VM, but I'm not sure.

 

I've upped the spec on the first post to 64GB and I'll check out prices on more than, depending on budget. I've got about 6.5k - 7.5k to spend on hardware alone (inc UPS). Then I've got to get the licensing in order with CALs etc.

[PotNoodleTech]

Spec looks good, If you're using 2012r2, might want to consider "using storage spaces with tiered storage mirror" for the mass storage as you could fit an additional pair of SSD drives in which will give you read and write caching and SSD automated storage for "high usage" data. The negative point is you will lose 50% of capacity in a mirror. So really it s a choice between - do you go for a big speed boost in daily usage (storage spaces mirror), or do you go for out and out capacity (Raid5/6). Also use dedupe to reduce your storage requirements.

 

Also consider dual UPS (one powering each power supply on the server) so that if a UPS fails the server keeps running. If you can't afford 2 x 1500 then maybe 2 x 1000 would be ideal.

 

However you do it - its a good spec for a primary school.

[hallb15]

Definitely increase the RAM. We recently bought a new server with 64GB of RAM thinking that would be plenty but I'm already looking to upgrade it. I've got 4 2012 R2 VMs running, one of which is SIMs, and a Foldr VM. At times its maxing out the RAM.

Do also carefully consider your storage as its harder to increase later...

[tmcd35]

Do also carefully consider your storage as its harder to increase later...

 

That's one of the reasons for suggesting storage spaces. So long as you have the SATA/SAS ports and free bays (or indeed just replacing physical drives with higher capacity) it is very easy to increase space later.

 

So really it s a choice between - do you go for a big speed boost in daily usage (storage spaces mirror), or do you go for out and out capacity (Raid5/6).

 

That's the other reason for suggesting storage pools. SATA drive are cheap compared to SAS drives. You can get the benefits of RAID1 with the capacity of RAID5. Use Tb+ enterprise SATA disks in a storage spaces mirror and use an SSD cache for extra speed boost.

 

I did some fag paper maths and came out at about £1000 cheaper than the SAS drives!

[PotNoodleTech]

That's one of the reasons for suggesting storage spaces. So long as you have the SATA/SAS ports and free bays (or indeed just replacing physical drives with higher capacity) it is very easy to increase space later.

 

Indeed, also you can very easily massively increase your storage capacity by adding a 6GB/s SAS HBA card into your server and then using SAS JBOD Trays to add more storage in batches of 12/24/48 drives depending on the tray - you can also daisy chain multiple trays for absurd numbers of drives. Storage Spaces really is so expandable and future proof.

 

That's the other reason for suggesting storage pools. SATA drive are cheap compared to SAS drives. You can get the benefits of RAID1 with the capacity of RAID5. Use Tb+ enterprise SATA disks in a storage spaces mirror and use an SSD cache for extra speed boost.

 

I did some fag paper maths and came out at about £1000 cheaper than the SAS drives!

 

Yeh not 100% sure about that - you can definitely drop to 900GB 10K SAS drives in order to leverage more space but I personally would only use SATA for less important storage, e.g. videos and whatnot, not for running the entire school.

 

But as mentioned - that's what storage spaces gives you, choices in speed versus capacity versus reliaiblity, choices in expandability vs cost, and all for free. People buying NAS's and SANS are missing a BIG trick.

[craig-schools]

So maybe using storage spaces I'd be better with a mix of

2x 2.5 240gb SSD £196

2x 2.5 600gb 15k £287

2x 2.5 900gb 10k £266 (prices from scan.co.uk for comparison)

 

but would this actually work as im only going to be putting HyperV VM drives on it and not "actual files" (like the most accessed word docs etc)? I assume it would move the entire VM drive container which would move my FILES VM to the SSD and put my PRINT VM on the 10k as an example? Its really difficult to get my head around the whole Storage Spaces without having a device to "play" on.

[tmcd35]

In that set up you should only need 1 SSD for the cache. Make sure its an enterprise model like the Intel S3700 to cope with a high volume of writes. Also, I don't think storage pools tiers the SAS drives, so it won't take into account the speed difference between drives. Personally I'd go for 4x2.5 900gb 10k in that set up. Should give you 1.8Tb mirrored plus the 240Gb cache, so 2Tb total.

 

You can then easily add extra SAS drives later on if you need more space.

[PotNoodleTech]

When you create a tiered storage, you decide if its simple (Raid0), Mirror (Raid 1), Parity (~Raid 5) for the whole tier (affects SSD and HDD together). The only one you want to use is Mirror. So you will need 2 or more SSD and 2 or more HDD.

 

You can't mix and match HDD's in the HDD pool, or SSD in the SSD pool - so you need to decide either 10k or 15k HDD you can't have both and either Intel S3500 or 3700 you can't have both.

 

One way around this could be to create multiple separate tiers. So one tier with 2 x SSD and 4 x 10K Drives and a separate tier with 2 x SSD and 4 x 15K drives. Because of the expense of the SSD - it doesn't really make sense. Because most of your data is going to be read and write cached on the SSDs (I've seen 90%+ SSD hit rate on some storage spaces blogs) then really 15k drives are not necessary. I'd go for bigger, 10k drives. in fact I am going for bigger 10k drives.

 

The way the cache works is it moves blocks of data automatically back and for to the SSD and the HDD. IT is data agnostic so these blocks can be small chunks of HyperV VMs for example if the whole school starts accessing a set of files in one particular HyperV VM - the system will overnight move those blocks onto the SSD tier. You don't need to do anything, and it's transparent to you and the end users (and the VMs).

 

You can also pin a whole VM to the SSD tier if you want, for example you could pin the VM that serves my documents files, to SSD for ultimate speed, and pin the VM that serves photo/video shared area to the 10K drives as ultimate speed is not needed.

 

It's a learning curve alright compared to raid and I would not suggest going ahead without a fair bit of testing.

[dhicks]

I've got about 6.5k - 7.5k to spend on hardware alone (inc UPS).

 

You could just get 6 1TB SSDs drives at £300 each, so under £2,000 in total, and not have to worry about performance.

[craig-schools]

I wasnt sure how advanced tiering was with regards the speed difference, hence the 2x SAS speeds. I'm already using 1.1TB of my storage I'm looking at at least doubling my available space so 2.4TB as minimum really. I could probably get away with 2TB with the 1x SSD and 4x 900's but I don't really want to find that in 1 - 2 years I need to spend another 1k on drives because they are full. I could up the 4x to 6x but that will be budget dependent.

 

I'll have it onsite about 3-4weeks before it goes live so should give me some time to play, I will be spooling up 6x of the 8x VM's during this time but they won't be in use, just a bit of prep work.

 

dhicks - 1TB SSD for £300, you point me to a server grade SSD for that price and I'll consider it. Found consumer ones for around that price point.

[PotNoodleTech]

Out of interest - you specify 8 x HDDs, is that because it's a 1U server?

 

I would strongly suggest going for a 2U or above in order to fit 12-16 2.5" drives in? That way you have easy future expansion even if you don't use them all right now.

[craig-schools]

No it's a tower unit so would have plenty of space for drives.

[PotNoodleTech]

Just to wet you're appetite here's what a tiered storage spaces test server (2009 model) looks like with 2 x cheapo SanDisk ssd and 4 x 10K SAS drives.

 

I'm looking forward to getting my hands on a 2015 model with 6 x 200GB enterprise SSD and 6 x 900GB 10K SAS drives!

 

[dhicks]

1TB SSD for £300, you point me to a server grade SSD for that price and I'll consider it. Found consumer ones for around that price point.

 

I've just been trying to get our HP Proliant DL360 Gen8 set up as a Storage Spaces server today. I've had some difficulties with the disk controller, the built-in HP Smart Array p420i. This is a RAID controller, designed to present arrays of harddrives to the OS as single drives. Initially it didn't want to start up with four 120GB Kingston SSDnow 300 drives in, but a firmware upgrade seems to have sorted that out. We have the basic confiruration, with no extra RAM cache on the controller, which turns out means that the controller only supports creating 2 drive arrays. For Windows Storage Spaces, you don't want to pass RAID arrays through to Windows, you want individual harddrives. You can't do that with this controller, it can't pass "raw" harddrives through to the OS, it has to create arrays. No problem, you can create single-drive RAID-0 arrays (in the latest version of HP's management utility there's even a handy automatic function to make each drive into its own single-drive RAID-0 array), but of course you're limited to only 2 arrays.

 

Therefore, it turns out, I'm going to have to buy a RAM upgrade (at around £150) for the controller just to try out Storage Spaces. And a RAID controller isn't really ideal for use with Storage Spaces, really you want an HBA controller of some kind - also, it turns out, around £150. You might want to bear this in mind if you're buying a new machine - if you're aiming to use Storage Spaces with it, you need to make sure the disk controller is suitible.

 

One feature the HP p420i does support is SSD over-provisioning. If I understand correctly, this aims to increase the lifetime of your SSD by keeping some capacity back for replacing sections as they wear out. This might allow you to use cheaper SSDs instead of enterprise-grade ones. Also, as far as I can see, the whole point of Storage Spaces is that it is relisiant to failure - if a harddrive fails you chuck it out and stick a new one in, the system carries on without pause. A bit wasteful of harddrives, but if consumer SSDs are a third the price of enterprise ones and last half as long you're still ahead.

[dhicks]

Just to wet you're appetite here's what a tiered storage spaces test server (2009 model) looks like with 2 x cheapo SanDisk ssd and 4 x 10K SAS drives.

 

Ooh - interesting. Just for comparison, I looked up a thread from December 2013 where I was looking at our then-new file server. In one post there I ran CrystalDiskMark on our file server:

 

http://www.edugeek.net/forums/windows-server-2008-r2/128270-windows-server-2008-r2-file-server-performance.html#post1096575

 

The results on the left-hand image are for smaller amounts of (100MB) data, small enough for the RAID controller (which has 1GB of RAM, I think) to cache, so the read and write speeds you see there are really the speeds for reading and writing RAM. Increase the amount of data (1000MB) and you see the underlying speed of the physical harddrives (4TB SATA models). I think we agreed at the time I'd made the right choice in RAID controllers, and over the past 18 months performance has been fine - 100-ish simultanious users on our network, all with all their files on the same storage array, doesn't cause problems.

 

Your results above, though, are for 1000MB of data, and show read and write speeds pretty much comparable to our RAID controller's RAM cache, but for actual persistant disk storage! Can I check - does your test server have a RAID controller with a cache, or an HBA card that passes through raw harddrives to the OS? What precentage of the traffic is handled by the SSDs - what's the speed / volume trade-off like? Any chance you can try with some 2TB or similar SATA drives?

 

One thought brought on by reviewing those results from 2013 - the sequential read speed for my server is slower than the sequential write speed. Is that because this is a cached device, so there's a cache-miss overhead every time the machine does a read operation? If the RAID controller with a cache is taken out and replaced with a bunch of raw disks, with SSDs being used as the highest level of disk cache in the system, is there a (possibly quite slight) increase in performance - is the RAID controller just getting in the way?

[craig-schools]

thats some good speeds there. I'll run a test on mine, I imagine they are fairly horrendous tho