Building a new SSD based file server

[twin--turbo]

SSD use in servers is not just down to the drives,

 

Replacing SAS with SSD in your old shelf will work, have a bit of performance upgrade but is susceptible to the killing of SSD as mentioned, and the performance will not make best use of the SSD.

 

PROPER SSD San Controllers for raid will both maximise the performance of the SSD and increase the lifespan of the drive.

 

Hence they cost a $%^$& fortune.

 

Rob

[tmcd35]

The price difference is a *lot*. Intel 320 600GB is about £500. A HP enterprise 600GB is about £2,000.

 

Interesting price for an Enterprise Grade SSD! Ouch!

 

Here's the thing though, a 600Gb SAS2 drive is £250 (WD XE 600GB 2.5" SAS Internal Hard Drive (WD6001BKHG) - www.misco.co.uk). Put in a decent RAID which is going to be faster, more reliable and cheaper? the SAS or the SSD?

 

I'm looking at SSD's now for desktops and laptops. It's part of my min. spec for new machines, but servers... Still don't think the time is right. Even at the other end of the scale I'm sure WD Velocoraptor's or SSD Hybrids would be a better bet.

[localzuk]

Interesting price for an Enterprise Grade SSD! Ouch!

 

Here's the thing though, a 600Gb SAS2 drive is £250 (WD XE 600GB 2.5" SAS Internal Hard Drive (WD6001BKHG) - www.misco.co.uk). Put in a decent RAID which is going to be faster, more reliable and cheaper? the SAS or the SSD?

 

I'm looking at SSD's now for desktops and laptops. It's part of my min. spec for new machines, but servers... Still don't think the time is right. Even at the other end of the scale I'm sure WD Velocoraptor's or SSD Hybrids would be a better bet.

 

Faster? The SSD by far. More reliable? Unknown at the moment. Cheaper? SAS.

[zag]

My experience on all the clients we run in school is the SSDs have been far more reliable than hard disks.

 

Our limited experience of SSD's in servers. One heavily used webserver and one Sims server has also been good over the last few years. Both using Intel X-25m drives. The web server particularly had about a 2000% increase in mysql speed. Amazing and surely the future.

[Willott]

Out of interest - CrystalDiskInfo - Software - Crystal Dew World somewhere within that it should give you a drive life remaining - would be interested to see on your SIMs server.

 

Cheers

 

Will

[Arthur]

I would not use them in RAID.

You just need a RAID controller that is designed to work with SSDs.

 

http://s15.postimage.org/ufcnmrxpn/Echostreams_Flache_SAN2_Bays.jpg

[Willott]

You just need a RAID controller that is designed to work with SSDs.

 

http://s15.postimage.org/ufcnmrxpn/Echostreams_Flache_SAN2_Bays.jpg

 

Build it yourself, use ZFS based OS... From the brief read I just did it seems to be fairly standard components, no hardware raid (though custom case and drive bays from what I've seen recently), will include a worked on OS, but I reckon I could put something similar together...

Edited October 16, 2012 by Willott
(hardware)

[zag]

And some more with another tool

 

 

Both the main ssd servers have been running for nearly 3 years! Amazing how time flies:)

Edited October 16, 2012 by zag

[synaesthesia]

*homer simpson drool*

[zag]

You just need a RAID controller that is designed to work with SSDs.

 

 

Gosh that is server porn! Want!

 

[zag]

Also found this great quote from another forum

 

Yes, well a good SSD will last a long time. That does not mean there aren't SSDs that won't last long. In fact, the OCZ Agility is a good example of how to build an SSD that won't last long.

 

Firstly the difference between the Vertex and Agility was that the Agility used cheap NAND compared to the Vertex. Performance wise the difference is not big, but I wouldn't be surprised if it cut the lifetime of the Agility in half compared to the Vertex drive.

 

Then there is the Indilinx controller. It was never as good as the Intel controller; it suffered from higher write amplification so it would always kill itself faster than the Intel would.

 

To add insult to injury there were those GC firmware updates that aggressivly kept up performance even without TRIM. What OCZ failed to mention was that those idle-time GC routines were flash killers. All the time your drive was idling it would sit there on its own wearing out its flash. It was so bad that in later firmware updates OCZ has rolled back most of those changes. But before that a lot of extra wear was done.

 

So while you can expect 15-20 years from an Intel X25-M, don't expect the Agility to match that. Also, the extra 20GB on the Intel drive matters. SSD endurance modeled on drive size is an exponential curve.

 

 

The drive wear indicator will reach 0% when the average write cycles to the flash reaches its rating (probably 5000 p/e cycles). But that rating is a minimum rating. You can normally expect to get 2-7x write cycles over the minimum rating. Your Agility drive still has some time left.

 

SSD wear (CrystalDisk Info drive health) - Overclockers Forums

[Guest Guest]

SSD's have longer lifespans that hard disks from my experience (I run over 200 of them in school). Just as an example I've had a webserver running for about a year and a half which gets over 150,000 hits a day. Not one hint of trouble (yet).

 

It's writes which are the problem, not reads. Unless your webserver something like facebook you aren't going to see many writes at all, probably less than a personal desktop PC. 200 is a school - in what, desktops or servers? Desktops, again you aren't going to see many writes at all, just initial imaging as documents/etc are on servers. Servers, unless it's a file or DB server again I can't see it doing many writes.

 

I'm toying with the idea of having an SSD based profile server, which would be cheap and fast. Reliability, however, is the main issue I see with SSDs in a heavy write environment.

 

One way I've heard suggested to help offset the lack of TRIM in a RAID array is to underallocate each disk in the array. How well it works I can't say.

 

 

 

And some more with another tool

 

Both the main ssd servers have been running for nearly 3 years! Amazing how time flies:)

 

4tb of data written is nothing. Depending on your file server you could write that much just doing the initial data copy. Be careful about applying your success in web servers to file servers.

Edited October 16, 2012 by Guest

[synaesthesia]

God, if the Agility lasts half as long as the Vertex which fails in minutes.... I hope they hurry up and go bust. They should have taken the chance to be bought out when they had it.

[Willott]

Just as a play about after the above super SAN... Anyone want to build a 5TB (suggest 2.5TB Usable) SSD SAN for ~10k in 2U? Spec list below for those interested:

 

Supermicro CSE-216A-R900LPB 2U Chassis

Supermicro X9DAi Motherboard

2x Xeon E5-2670 8 core 2.6GHz CPU

2x Xeon Heatsinks (Supermicro ones)

16x 16GB PC3-10600 ECC DDR3 RAM

2x 120GB Intel 520 SSD (Mirrored Boot)

22x 240GB Intel 520 SSD (suggest ZFS pool of mirrored vdevs)

3x LSI 9207-8i HBAs

2x Intel X520-DA2 10GbE (or swap out for something higher...)

 

Could always look to throw a small Fusion IODrive2 in for write caching... and upgrading to 16x 32GB sticks of RAM for bigger ARC... Use a ZFS based OS (Nexenta, Openindiana, Freenas, etc), away you go.

 

But anyway... The other thing to consider is cost of replacement devices - could you buy a replacement SSD every 2 years (for example - depends on load on server and write cycles, may need to be each year) for cheaper than an enterprise device, clone your server across to the new disk and run off there (considering you're using DFS, you could happily take a server down for an hour whilst cloning...). Do a full wipe on your removed disk and stick it in a desktop around site to work out it's remaining life. And as technology advances, as you replace with a new device, you may have a bigger or more reliable device when you buy in 2 years time to replace the current one... just spitballing an idea there...

 

Of course, you could look to build a shared storage device, store your files and VMs on there, and use SSD and RAM caching (feel free to ask about this if you want more info) - which may be more expensive than building an individual server, but is the way we've gone.

Edited October 16, 2012 by Willott

[Willott]

One way I've heard suggested to help offset the lack of TRIM in a RAID array is to underallocate each disk in the array. How well it works I can't say.

 

Apologies for double posting - http://cache-www.intel.com/cd/00/00/49/23/492354_492354.pdf page 12 suggests there may be a 4x increase in write endurance for 20% over provision. Over provisioning does seem to be one way to significantly extend the life of write heavy workload SSDs.

[zag]

If you notice the screenshots I posted earlier, both our 3 year old X-25 SSD servers are not running trim (it wasn't available at the time). And have had no problems with performance levels at all.

 

As far as I can tell they are as quick as the day we installed them.

[localzuk]

If you notice the screenshots I posted earlier, both our 3 year old X-25 SSD servers are not running trim (it wasn't available at the time). And have had no problems with performance levels at all.

 

As far as I can tell they are as quick as the day we installed them.

 

But as has been mentioned - you're using them for mainly read operations aren't you? Its writes that kill drives. If you put those drives into a (high write) SQL server, or a fileserver, they'd be dead in no time.

[Willott]

But as has been mentioned - you're using them for mainly read operations aren't you? Its writes that kill drives. If you put those drives into a (high write) SQL server, or a fileserver, they'd be dead in no time.

 

Which is why I suggested checking out the amount of data written on current file server on a daily basis (or for the past month). This will give a rough idea of the estimated life of the drive. Also, with over provisioning, you could extend the life. Just trying to find something that will give stats on windows file server for data written.

[zag]

But as has been mentioned - you're using them for mainly read operations aren't you? Its writes that kill drives. If you put those drives into a (high write) SQL server, or a fileserver, they'd be dead in no time.

 

Very good point. The web server particularly is massively read only biased. Sims server probably as well. I do use a 300GB SSD on my home pc which has a massive write and read throughput though, in fact a scary amount of data has gone through that over time . Not had a problem with it for 2 years though.

 

I think the basic point i'm hoping to confirm is that SSD drives simply don't "wear out". They are much more likely to die from a firmware bug or controller failure than write cycle wear and tear.

 

I had a look at our Fileserver backup logs and it typically increases by around 2gb a day. That really isnt much I don't think.

[Willott]

Very good point. The web server particularly is massively read only biased. Sims server probably as well. I do use a 300GB SSD on my home pc which has a massive write and read throughput though, in fact a scary amount of data has gone through that over time . Not had a problem with it for 2 years though.

 

I think the basic point i'm hoping to confirm is that SSD drives simply don't "wear out". They are much more likely to die from a firmware bug or controller failure than write cycle wear and tear.

 

I had a look at our Fileserver backup logs and it typically increases by around 2gb a day. That really isnt much I don't think.

 

They do wear out - however, you can keep a close eye on how long they have left (you could probably set up a SMART monitor to check the life remaining smart code). It all depends on the write levels you're doing.

 

2GB a day is how much your backups are increasing - that doesn't necessarily take into account changes to files which don't increase in size or files that increase in size, but are rewritten (or the way the read/write cycle works - Windows may rewrite the entire file, not just write the additional blocks). Still trying to find a suitable utility to give a clear picture on actual write load per day...

 

But even saying that, you're probably nowehere near a full disk write per day, so I can't see if being an issue.

Edited October 16, 2012 by Willott
added final sentence

[CyberNerd]

We ran Vertexes on a Citrix Cluster of 10 servers for 2yrs. The citrix servers get a fair amount of writes due to their nature. Only had one failure in that time and the smart looked pretty good on the others up until we scrapped them. We upgraded them this term with Icore i7-3820's, and naturally Intel SSD's.

[Willott]

Found some possible software, just testing out SsdReady: Life measurement and monitoring tool for solid-state drives which seems to monitor writes - leave it running for a couple of days on your fileserver and it should give you a decent estimate of actual writes. Currently running it on our student file server out of interest...

[ICTLTD]

I think the misconception on SSD reliability is because they don't fail (in general) like a mechanical drive instead their performance and capacity dwindles with write loads. We have a 3 year old laptop here that has a SSD and that is operating at 50% health and that as hardly done much in the way of heavy writes. It still works, but I would not say that SSD is in general more reliable, they just die a slow death rather than the mechanical drive heart attack.

 

There is a reason why Dell, HP etc only supply enterprise SSD not DT SSD in their servers, if they thought they would be reliable enough they would supply them as there would be significant price advantage.

The truth is they are not classed as a reliable solution when there are high rates of writes, if they sold them they would have to support them so they are not economically viable in terms of warranty for servers.

If manufacturers think DT SSD is not economically viable in servers, the chances are they arn't. They predict (and have presumably tested to death) that DT SSD will have short life spans in an enterprise environment.

 

Unless you need a high number of IOPS I would recommend 15K SAS with raid 5 (or greater) for file servers. If you want to go SSD (preferably enterprise class) just keep it for the OS on the server (Mirrored) but I would keep the file read write side as SAS. I can't say we have seen a school that needs anything currently greater than 15K SAS for file servers, not storage Vs cost Vs performance anyway.

 

SSD is good for fast access reads, low amounts of writes.

SAS is good for reasonably fast access and heavy read/writes.

Edited October 16, 2012 by ICTLTD

[Willott]

Dell, HP etc supply enterprise SSDs because they can charge a premium for them, and they reduce the likelihood of further support requirements. There is no reason they couldn't supply "desktop" SSDs with limited warranties and overprovisioning, apart from the fact that they can't charge such a premium. Taking the write limitation out of the equation:

 

Generally speaking (SSD being desktop SATA 3.0 ie Intel 520)

 

Write IOPs SSD > Write IOPs 15K SAS

Read IOPs SSD > Read IOPs 15K SAS

Write throughput SSD > Write throughput 15K SAS

Read throughput SSD > Read throughput 15K SAS

Price for 480GB SSD is only slightly more expensive than a 15K 300GB SAS (20% over provision the SSD to get better endurance and better performance over time)

 

Again, it all depends on the actual write figures. I don't think many of us on this forum have a true "heavy" write situation, we're not running constant video editing, we don't have massive oracle databases with constant transactions, we're not talking about saving CCTV images onto it, we're not talking Facebook database servers, we're talking day to day file data... for 12hrs to the day there'll be little to no writes (that big bit overnight when there are no staff or students in school, and those staff working off site won't be doing massive amounts of writes). I have a feeling if we all looked over a period of a few days, we'd see we probably don't hit more than 20GB writes per day (and if you do, then consumer SSD probably isn't for you... unless you over provision).

 

But, without knowing write characteristics, no further comment can be made to the specific case.

[ICTLTD]

Dell, HP etc supply enterprise SSDs because they can charge a premium for them, and they reduce the likelihood of further support requirements. There is no reason they couldn't supply "desktop" SSDs with limited warranties and overprovisioning, apart from the fact that they can't charge such a premium. Taking the write limitation out of the equation:

The warranty is limited on Enterprise SSD anyway...(gulp)

But if you look at the Dell/HP etc price and the cost of enterprise SSD's as a component on the web I do not really think they inflate the price it is the fact they are comfortable they are up to more extreme demands. (also including Enterprise SSD does not increase the warranty cost)

 

But the main thing is the performance needed? In reality if you need something faster than 15K SAS, SSD is the only option really. If you don't need faster than 15K SAS I would say 15K SAS is the better proven option and is an enterprise level solution and cheaper in general / long term.

 

But there will always be instances where SSD may be the more suitable option, in DT virtualisation for example where the IOPS requirement is higher than storage requirement it may work out cheaper.

 

Best thing is to give it a go and see how it works out it may be suitable, but I see the main stumbler as warranty. Over a 5 year life cycle if you do wear out your SSDs due to writes or they start degrading you have to replace them where with SAS you can have them covered under warranty no problem.

Until SSD has the same warranty covering by manufacturers as SAS it is always going to have a limited amount of risk attached IMHO as you never know when things might change.

Edited October 16, 2012 by ICTLTD