Building a new SSD based file server

[CyberNerd]

Appropriately timed discussion on slashdot about

Ask Slashdot: How Do SSDs Die? - Slashdot

[Arthur]

Found some possible software, just testing out SSDReady

If you need something that is much more comprehensive, try hIOmon.

 

http://i.imgur.com/LLv4p.png http://i.imgur.com/5y2PS.png

 

http://i.imgur.com/O4BuV.png

[ijk]

I think your money is better spent elsewhere. The performance, longentivity and other merits (price?) of particular SSDs are something to consider but is this a good idea for your school? Would the money be better spent on, for e.g. a system to reliably backup the data you are currently storing?

[zag]

That slashdot article kind of sums up my views as well.

 

First time accepted submitter kfsone writes

"I've experienced, first-hand, some of the ways in which spindle disks die, but either I've yet to see an SSD die or I'm not looking in the right places. Most of my admin-type friends have theories on how an SSD dies but admit none of them has actually seen commercial grade drives die or deteriorate.

 

I would think power failure/surges is the biggest risk of a good SSD failing.

Edited October 17, 2012 by zag

[tmcd35]

I'll admit that this thread as altered my attitude/perception towards SSD's some what, but I still think it's a case of the right tool for the right job. I really can't see SSD's, currently, being the right tool for the job in the case of a file serve in a school environment. It's good to know that in an average home environment a good SSD could last 13-14 years worth of writes, but in a school? Got to be a fraction of the time...

 

For my money standard hard disks are still where it's at for a file server (preferably 15k SAS). Want more speed? Buy more spindles!

[localzuk]

I'll admit that this thread as altered my attitude/perception towards SSD's some what, but I still think it's a case of the right tool for the right job. I really can't see SSD's, currently, being the right tool for the job in the case of a file serve in a school environment. It's good to know that in an average home environment a good SSD could last 13-14 years worth of writes, but in a school? Got to be a fraction of the time...

 

For my money standard hard disks are still where it's at for a file server (preferably 15k SAS). Want more speed? Buy more spindles!

 

My attitude is, as you say, right tool for the job. We're having a hybrid solution for our storage needs. Namely 9 SAS drives for 'file storage', and 16 SSDs for virtual machines and profiles.

 

True, I likely would never have gone this route if I had chosen all of it, rather than just having to get what was left for me working.

[CyberNerd]

Like ICTLTD said "the main thing is the performance needed?"

probably not In a file server - but SSD's make a dramatic improvement to Terminal Servers. So Like Localzuk we have a happy 'medium' (pun intended) of SSD's, SAS and SATA.

[Willott]

Oh, other thing I forgot to put on previous post:

 

Power requirements for SSD < Power requirements for 15K SAS HDD

 

If you want higher performance from SAS, add more spindles, which requires more physical space and more power (and more cooling and a noisier system). In some ways, I'm tempted by building several ITX boxes with SSDs and have multiple in a DFSR namespace, low powered boards and SSDs, they're only file servers, so the CPU requirements are fairly insignificant... a lot cheaper than full enterprise solutions, and you get your reliability, stability and in some cases performance from your increase in devices - someone earlier said they ran a 10 server Citrix farm based on desktop components - was this because desktop components were cheaper? Could the farm handle outages better than a farm built of 3 enterprise class devices (which is what you'd probably get for the same cost) and was the performance as good or better?

 

For the same cost as an HP server with Enterprise SSD, you could probably build 4 mini itx boxes, with desktop SSD, 2 mini itx (and you'd still be using less power than the HP box, and you can get 2 mini itx in a 1u case), for HP with SAS you'd probably be able to do 2 mini itx with SSDs, much quicker access, less power.

 

You may not need the performace, but if you can have the performance for the same cost and put in place other things to assist with reliability and the other issues brought up so far, why not have it? Who's to say that in a year's time, the school will go BYOD requiring access for all students to the file server, with several hundred random IOs per second as students access and write their files? We have a mix of all devices here (SAS HDD, SATA HDD, SSD), the only place I'm looking at buying new drives, I look to either SSD, or for large requirements 7.2k SAS - using SSD/RAM to cache.

 

I'm just going to flick back to the FlacheSAN2 device - aparently they consider prosumer SSDs (intel 520) suitable for enterprise environments - you're not after all going to put a device with that spec in a house - so why shouldn't we?

[ICTLTD]

Like ICTLTD said "the main thing is the performance needed?"

probably not In a file server - but SSD's make a dramatic improvement to Terminal Servers. So Like Localzuk we have a happy 'medium' (pun intended) of SSD's, SAS and SATA.

I think you and Localzuk are right with the implementation. SSD is ideal for Terminal services / DT virtualisation, SAS Tier 1 storage, Sata or near line SAS for low performance storage/archive.

 

If you use the different technologies to their strengths they all have a level of performance per £ spent where they are winners.

[localzuk]

Like ICTLTD said "the main thing is the performance needed?"

probably not In a file server - but SSD's make a dramatic improvement to Terminal Servers. So Like Localzuk we have a happy 'medium' (pun intended) of SSD's, SAS and SATA.

 

Oh, and we use SATA for our backup server. No need for uber-speed on that!

[Willott]

I think you and Localzuk are right with the implementation. SSD is ideal for Terminal services / DT virtualisation, SAS Tier 1 storage, Sata or near line SAS for low performance storage/archive.

 

If you use the different technologies to their strengths they all have a level of performance per £ spent where they are winners.

 

Yep, but when do we then break from - oh, we may as well just use SAS because that's what's always been used in this situation... as more people start to use SSDs and as SSD technology progresses (to be honest, I think we're at tipping point now), the price of SSDs will come down, and the other benefits of SSDs will start to be more apparent (power savings add up over time, less noise and vibration, less heat released so less cooling required).

 

I see no reason why the following cannot happen now:

 

Low-medium write requirements - prosumer SSD

High write requirements - enterprise SSD (or RAM SSD as cache to mass storage)

Mass storage - NL-SAS or SATA (dependant on other components in system)

 

Hopefully a larger shift to SSD will then force both HDD manufacturers and SSD manufacturers to reconsider their pricing and the technologies they use. Until electron spin storage becomes commercial, why don't we look at ways to shift the demand to SSDs and in doing so cause a drop in both HDD prices (large supply, small demand, so price has to decrease) and increase the pressure on SSD manufacturers to improve both storage speed, latency and reliability and increase capacity.

[ICTLTD]

You may not need the performace, but if you can have the performance for the same cost and put in place other things to assist with reliability and the other issues brought up so far, why not have it? Who's to say that in a year's time, the school will go BYOD requiring access for all students to the file server, with several hundred random IOs per second as students access and write their files? We have a mix of all devices here (SAS HDD, SATA HDD, SSD), the only place I'm looking at buying new drives, I look to either SSD, or for large requirements 7.2k SAS - using SSD/RAM to cache.

 

I would say the main thing there is warranty, you can put in fault tolerance but on critical things (Like a file server) can you risk things not being replaced/fixed next day or not being able to get replacement identical MB's or disks a year later?

Main thing in enterprise level solutions is getting the performance requirement right, but nearly equal is the reliability and support. Make an error on judgment on that and you could be increasing your costs in terms of time taken to support issues, reconfigurations due to failures etc.

Although more expensive I would try to keep with enterprise it where money allows it as you can be fairly sure the solution will be robust enough to last 5 years +.

[pcstru]

I see no reason why the following cannot happen now:

 

Low-medium write requirements - prosumer SSD

High write requirements - enterprise SSD (or RAM SSD as cache to mass storage)

Mass storage - NL-SAS or SATA (dependant on other components in system)

 

For high IOP requirements, you can buy prosumer SSD's, simply replace them every year and still be quids in against a high IOP SAS setup. Why : because the performance difference is 20x but the cost difference between a large prosumer SSD and a 15K SAS is ... well, the SSD will often be cheaper. And in a years time the SSD's will be even cheaper/bigger.

[Willott]

For high IOP requirements, you can buy prosumer SSD's, simply replace them every year and still be quids in against a high IOP SAS setup. Why : because the performance difference is 20x but the cost difference between a large prosumer SSD and a 15K SAS is ... well, the SSD will often be cheaper. And in a years time the SSD's will be even cheaper/bigger.

 

This is the point I've been trying to make - and stick in other bits to make your system reliable - yes it's potentially a little more support work on your side, but if you have 4 devices sharing the load, and 1 goes down, it's less high priority than if you have 2 and 1 goes down. You can fix it next holiday, you only up the priority if 2 devices go down...

[localzuk]

For high IOP requirements, you can buy prosumer SSD's, simply replace them every year and still be quids in against a high IOP SAS setup. Why : because the performance difference is 20x but the cost difference between a large prosumer SSD and a 15K SAS is ... well, the SSD will often be cheaper. And in a years time the SSD's will be even cheaper/bigger.

 

Depends if you think your time having to swap things about is worth less than an enterprise SSD. Personally, I'd want a solution I can just leave and ignore until it is due for replacement in 3-5 years.

[ICTLTD]

Yep, but when do we then break from - oh, we may as well just use SAS because that's what's always been used in this situation..

 

I would say as I said before (at a server level) when manufacturers offer the same warranties as SAS. When you know you can get a disk within 4 hours without jumping through hoops finding out how many writes you have had or being forced to buy enterprise discs a 4X the costs.

At the moment 15K SAS is an enterprise level solution, DT SSD is not (not at a manufacturer level anyway) so comparing a 15K SAS to a DT SSD is not a great comparison at the moment, enterpise SSD is more expensive for a reason.

 

I am happy that things are evolving but I would always recommend having a well supported backbone to any server room. The day large manufacurers adopt low cost SSD in enterprise and support it without issue is a good day to compleately switch in my opinion.

Until that point use the solution which covers your requirements and is most cost effective to do that. At the moment I would say for SSD that is DT virtualisation/terminal services.

[pcstru]

On a raid set-up, I'd just be pulling a drive, allowing rebuild and then move on to the next. I guess I might have to order the parts, but my time would need to be seriously expensive to become a major consideration (I'm comparatively well paid from what I can gather). And that's assuming you are running writes at a rate that uses up your drives within the 3 years - which is not a given.

 

ETA - Just to be clear I'm looking at this from the point of view of being able to deliver a high IOPS solution. Even with SAS drives getting into a few thousand IOPS throughput on (say) RAID starts to require a lot of physical drives. For things like user file storage I'll be using large SATA drives, I'd guess for a good while yet.

Edited October 17, 2012 by pcstru

[localzuk]

On a raid set-up, I'd just be pulling a drive, allowing rebuild and then move on to the next. I guess I might have to order the parts, but my time would need to be seriously expensive to become a major consideration (I'm comparatively well paid from what I can gather). And that's assuming you are running writes at a rate that uses up your drives within the 3 years - which is not a given.

 

But you've said Prosumer SSD, which means lack of RAID support. So that wouldn't work.

[Willott]

But you've said Prosumer SSD, which means lack of RAID support. So that wouldn't work.

 

No... lack of TRIM support in RAID, devices still have GC routines...

[ICTLTD]

I would say on RAID with SSD if your failure is due to writes and you have 3+ identical disks installed at the same time you are more likely to get more disk failures at around the same time due to them all reaching there usable life nearly all together, taking out the entire array.

It is not like a mechanical failure so a closer eye on disk health would be required to stop a complete array going down.

Edited October 17, 2012 by ICTLTD

[localzuk]

No... lack of TRIM support in RAID, devices still have GC routines...

 

Having seen some tests without TRIM in anything of a data writing situation, you're planning for disaster there to be honest. Performance degrades severely without TRIM.

 

I'll try and find it, but there was an article somewhere which showed the difference - they had 2 identical drives, one TRIM enabled, one without. Both worked well at first, (I think they simulated a high IO database on it), and then within a few hours, the non-TRIM one started slowing down, getting so bad that the device was basically useless by the end of the test.

[Willott]

Consider age of article and hence age of device/firmware - things have moved on. Also look at effect of over provisioning on SSDs (as I've mentioned before). Has a positive effect on write performance endurance.

[pcstru]

I would say on RAID with SSD if your failure is due to writes and you have 3+ identical disks installed at the same time you are more likely to get more disk failures at around the same time due to them all reaching there usable life nearly all together, taking out the entire array.

It is not like a mechanical failure so a closer eye on disk health would be required to stop a complete array going down.

 

Excellent point. I'm thinking of RAID from the point of view of building large logical volumes as much as failure. For our high IOP loads I really need a replicated system - but that too would be subject to the same write lifetime.

[pcstru]

But you've said Prosumer SSD, which means lack of RAID support. So that wouldn't work.

 

I'm not sure you can say it wouldn't work at all - (modern SSD's tend to have better garbage collection than in the past). I think it would need some careful selection of devices and management of the actual loading (ensuring plenty of available free space), but given the cost of provisioning high IOPS with SAS, it has to be worth a look. I can understand caution though!

[zag]

Having seen some tests without TRIM in anything of a data writing situation, you're planning for disaster there to be honest. Performance degrades severely without TRIM.

 

I've been using a Non-Trim SSD on our Sims server for nearly 3 years now. No slowdown yet.