New Server, Sanity Check on Spec

[ACrowe]

Personally I wouldn't go flash, 10k drives should be sufficient.

[PotNoodleTech]

Your results above, though, are for 1000MB of data, and show read and write speeds pretty much comparable to our RAID controller's RAM cache, but for actual persistant disk storage! Can I check - does your test server have a RAID controller with a cache, or an HBA card that passes through raw harddrives to the OS? What precentage of the traffic is handled by the SSDs - what's the speed / volume trade-off like? Any chance you can try with some 2TB or similar SATA drives?

 

Test server has a HBA card (Dell 6GB SAS) native pass through. The SSDs cache reads and writes which is why we see near raid card cache performance. It's only a test server a the mo and I've only just set it up so cant give many performance metrics. Don't have 2.5" 2TB drives to fit in sorry.

 

One thought brought on by reviewing those results from 2013 - the sequential read speed for my server is slower than the sequential write speed. Is that because this is a cached device, so there's a cache-miss overhead every time the machine does a read operation? If the RAID controller with a cache is taken out and replaced with a bunch of raw disks, with SSDs being used as the highest level of disk cache in the system, is there a (possibly quite slight) increase in performance - is the RAID controller just getting in the way?

 

It's probably because your raid card only caches writes, not reads so reads end up being reliant on the seek time etc of the hard drives. This is much slower. It's normal to get what you get.

 

If using storage spaces you shouldn't use a RAID controller as it gets in the way of what storage spaces wants to do, you should only use a HBA Pass-through type card.

 

Will hopefully be getting some VMs on there soon for some real world testing.

[SHimmer45]

so using a RAID controller causes an issue for SS's in terms of speed

[tmcd35]

I've been running some Crystal Disk tests on our two storage servers. In a like for like set up our older server, that uses disks presented to the storage pool as individual RAID-0 volumes, runs at about half the speed of our newer server that uses an HBA.

 

I've also noticed that the SSD cache on both systems is slower at reading than accessing the underlying drives directly but much faster at writing.

Edited May 12, 2015 by tmcd35

[dhicks]

It's only a test server a the mo and I've only just set it up so cant give many performance metrics.

 

Would giveing it a larger CrystalDiskMark test be any good - give it enough data that any caches get full, then we're looking at raw disk read / write speeds?

 

It's probably because your raid card only caches writes

 

Is that right? I'd assumed hardware RAID controllers cached reads, along with a read-ahead buffer, or is all that simply left up to the OS these days?

 

If using storage spaces you shouldn't use a RAID controller as it gets in the way of what storage spaces wants to do, you should only use a HBA Pass-through type card.

 

Any idea what happens if you use an SSD drive RAID array and a SATA drive RAID array as components in a Storage Spaces volume? Does it all go horribly wrong in some way?

[dhicks]

In a like for like set up our older server, that uses disks presented to the storage pool as individual RAID-0 volumes, runs at about half the speed of our newer server that uses an HBA.

 

That's quite a difference for something that I'd assumed would casue a small performance hit. Any idea why this ism exactly?

[tmcd35]

None at all. There is CPU and RAM difference between the two servers but I can't believe they are the cause of this effect. The hard disks are 100% identical in both arrays (2x100Gb SSD, 7x900Gb 10k SAS, 6x600Gb 15k SAS). The only major difference is the controllers. I've run the tests at 4GiB, 1GiB and currently doing 100MiB, all over 6 passes. The results remain consistent each time they run. I'm guessing there is something the RAID controller is doing in caching/managing the data that is getting in the way with storage spaces management of the drives. But the results are around half the speed on most tests with the RAID controller.

[PotNoodleTech]

Would giveing it a larger CrystalDiskMark test be any good - give it enough data that any caches get full, then we're looking at raw disk read / write speeds?

 

Ive tried that! Still fast!

 

 

[SHimmer45]

stumbled across this white paper published by Fujitsu haven't really read it yet but might prove interesting reading as there looks to be comparisons in different storage setups.

Fujitsu Windows-storage-spaces-performance paper

[tmcd35]

Thanks @SHimmer45 I was looking for something just like that! Scanning through the document and it appears to be confirming the speeds I'm seeing - which is a good thing!

[DMcCoy]

Don't forget the big issues with storage spaces beyond terrible parity and passable mirror performance.

 

1) The "good" features are only available with ReFS - Integrity

2) Integrity feature is not enabled by default for files because it has a massive performance impact

3) Integrity cannot be used for VHD/VHDX files on ReFS

 

All in, storage spaces and ReFS is a big disappointment.

[tmcd35]

Usual Microsoft affair. I'll stick with Storage Space now, early adopter and all that. It be fine in Server 2016

[PotNoodleTech]

Don't forget the big issues with storage spaces beyond terrible parity and passable mirror performance.

 

1) The "good" features are only available with ReFS - Integrity

2) Integrity feature is not enabled by default for files because it has a massive performance impact

3) Integrity cannot be used for VHD/VHDX files on ReFS

 

All in, storage spaces and ReFS is a big disappointment.

 

Maybe you should read that Fujitsu paper.

 

Meh I think you're oversimplifying those three negative points as they're all negative points of something which isn't neccasary to leverage the benefits of storage spaces - REFS is a whole different thing. We're talking predominantly about the "hardware comparison" of storage spaces VS RAID neither of which require REFS.

 

I don't view storage spaces as a big disappointment, I view it as very promising.

 

1. Raid like performance in Mirror mode (VS RAID 1 or 10 which lets face it, is the most commonly recommended and highest performing RAID level).

2. Automated READ/WRITE caching if you use SSD. FOR FREE. This is a massive performance booster for every school and you don't have to spend £40k on a SAN to get it.

3. Not tied in to any particular manufacturer of storage trays or disks like you would be with a SAN.

4. Not tied in to any nasty yearly warranty/support costs like you would be with a SAN.

5. More expandable than MOST SANS, and for free - you want more disks, slot more in? you need more slots, slot more trays on, you need even more? Fit more SAS HBA cards and slot even more trays on. It's absurd the kind of monster storage system you could build with a similar budget to what schools often spend on a SAN (£10-40K)

6. And here's the clincher for me, it's only going to get better with each release. We've had massive leaps from 2003-2008-2012r2 who knows where we'll go.

 

Or am I going mad it is possible after 14 years of this

[DMcCoy]

Maybe you should read that Fujitsu paper.

 

It's absurd the kind of monster storage system you could build with a similar budget to what schools often spend on a SAN (£10-40K)

 

And here's the clincher for me, it's only going to get better with each release. We've had massive leaps from 2003-2008-2012r2 who knows where we'll go.

 

2012 to R2 has no improvements for storage spaces.

 

Storage spaces (sometimes) breaks if you fill up the allocated space

 

NTFS Dedupe still has issues, moved a test server away from it with 2012 after file corruption with certain multi user files.

 

You can now have massive storage, with no usable integrity. Hyper-V will refuse to start a VM with integrity enabled!

 

MSDPM doesn't work with ReFS stored VMs (it pretends to backup, but does nothing).

 

 

 

It has all been steps in the right direction, but MS is too inconsistent to make anything useful out of it yet. They made a new filesystem, and made it unusable in the most common scenarios.

[craig-schools]

So if I use Storage Spaces I can't put my HyperV VM's on it? Surely thats not right...

 

I won't be storing any "files" directly on the Storage Space, only VM's which will then hold the daily usage word / excel etc docs.

[PotNoodleTech]

Yeh that's what I am doing. Storage Spaces running as NTFS volumes work just as normal as far as I can see.

[craig-schools]

Thanks for that clarification. It would be the clincher between using RAID or Storage Spaces if it messed with my VM's which is the whole point of this / any server build.

[matt40k]

That's quite a difference for something that I'd assumed would casue a small performance hit. Any idea why this ism exactly?

@tmcd35 - use cpuinfo to check out if the CPU supports Intel VT-d and then check if its enabled. Edited May 21, 2015 by matt40k

[SHimmer45]

[slightthreaddrift] its something id love to be able to test the differences between RAID / HBA etc but only having aging test equipment is a pita.

anyone purchase a smallish JBOD caddy for use with a HBA etc which doesnt cost the earth

[/threaddrift]

[SHimmer45]

@tmcd35

 

intelARK is your friend for CPU info without the need to run a polling tool on a server which is something im always very nervous about doing

[matt40k]

True, grad the cpu model from system and google it. Not sure about

polling tool on a server which is something im always very nervous about doing

 

Why would you be nervous about pooling server resources? You need to do it for monitoring, auditing\security, capacity management.

[SHimmer45]

goes back to a couple of odd experiences with CPU-Z on some older kit.

[dhicks]

Just to wet you're appetite here's what a tiered storage spaces test server (2009 model) looks like with 2 x cheapo SanDisk ssd and 4 x 10K SAS drives.

 

Right, so I've just managed to get our test server up and running with tiered storage - 2 x 128GB Kingston SSDs and 4 x 500GB 2.5" drives taken from laptops during upgrades, so not the most "enterprise" of drives. CrystalDiskMark gives:

 

 

This compares with our current productions servers, Dell machines with RAID cards with 1GB of cache RAM each:

 

 

And, for comparison, AButters' results for tiered storage from earlier in this thread:

 

 

Write speed on our array is rather poor. Thinking through why this is, I'm wondering if the benchmark is actually mesuring the correct thing. This is tierd storage, the SSD is not acting as a cache, so when (random, benchmark-generated) data is written it is written straight to the crusty, slow second-hand harddrives. Windows would only move that data to the SSD tier if it figured it was often used, which it isn't going to be with a synthetic benchmark. The figures above are telling me that SSD read speed is okay, which is nice, and that the other harddrives are slow, which I know already.

Edited May 28, 2015 by dhicks

[Arthur]

Write speed on our array is rather poor.

Is write caching enabled for the drives in Windows?

[dhicks]

Is write caching enabled for the drives in Windows?

 

Good point. For each individual drive, in Windows (set via the "Policies" tab for each harddrive from Device Manager), no. For the Microsoft Storage Space Device constructed from a composite of those individual disks, yes.

 

For the HP P420i RAID controller, for individual disks (well, individual 1-drive-sized disk arrays), no. I can change that via HP Smart Storage Administrator by setting the Physical Drive Write Cache State - new results:

 

 

That seems to even the figures out somewhat, although the stated read performance seems to have around about halved in some cases.

 

I think the issue is that this benchmark isn't testing what Storage Spaces actually does. I'll try a script that copies some files around, see if I can better make it mirror an actual workload.

Edited May 29, 2015 by dhicks