Building a new SSD based file server

[zag]

Our file server is coming up for renewal so I'm going through the plans of building a new one.

 

I was thinking one of the new 12th R320 Gen Dell rack servers. 2 x intel 600GB SSD drive and a low power xeon processor.

 

What operating system should I use and what features should I use within it?

Should I hyper-V it?

Anyone got any thoughts on SSD drives for web servers?

[gshaw]

Interesting, not seen anyone going for SSDs in a file server before. How much are the SSDs, I take it they're the server grade ones not desktop drives?

[RTFM]

QNAP TS-EC1279U-E10G/1.44TB-SSD 1.44TB SSD 10GbE 12.. | Ebuyer.com

 

 

I think I saw an SSD SAN which was 10tb and $100k when we were looking at ours haha

[zag]

Interesting, not seen anyone going for SSDs in a file server before. How much are the SSDs, I take it they're the server grade ones not desktop drives?

 

The 600GB SSDs are £600 each.

 

And no they are standard 320 series intel drives. I've been running lots of them and their reliability has been great so far, much better than hard disks. I'm doing this now as the 320's are going out of stock and the new 520s have sandforce controllers which I just don't trust

 

The idea of using SSDs would be to speed up user access, backups and hopefully increase reliability as well.

 

No one else made the jump yet?

[FN-GM]

increase reliability as well

 

Might not do. SDD have a limit of how many times your can read and write to them. I would have though a file server would hit this limit fast compared to other uses.

[zag]

Might not do. SDD have a limit of how many times your can read and write to them. I would have though a file server would hit this limit fast compared to other uses.

 

SSD's have longer lifespans that hard disks from my experience (I run over 200 of them in school). Just as an example I've had a webserver running for about a year and a half which gets over 150,000 hits a day. Not one hint of trouble (yet).

 

That statement is actually a bit of an old wives tale. I've never had an SSD run out from read/write levels. The vast majority of failures are from the controller. Just ask anyone with a OCZ drive lol

Edited October 16, 2012 by zag

[localzuk]

The 600GB SSDs are £600 each.

 

And no they are standard 320 series intel drives. I've been running lots of them and their reliability has been great so far, much better than hard disks. I'm doing this now as the 320's are going out of stock and the new 520s have sandforce controllers which I just don't trust

 

The idea of using SSDs would be to speed up user access, backups and hopefully increase reliability as well.

 

No one else made the jump yet?

 

Standard drives don't work with RAID properly - RAID stops TRIM working, meaning the drives degrade very quickly. So, you need to get enterprise grade ones.

 

We are waiting on a HP Gen8 380 - the storage edition with 25 USFF bays. We are putting 16 x 100GB HP SSDs in and 9 600GB SAS drives.

[glennda]

What connection out the other side are you going to have? Single 1GBE? If so then your going to hit the network limit for before the disks massively.

[zag]

I would not use them in RAID.

 

As far as I can make out RAID technology was designed to speed up hard disks and prevent complete failures of a single drive. Both of which don't apply to SSD's as much.

 

Assuming its a hyper-v machine I would just backup the whole thing very quickly. The backup strategy would obviously have to be much more full-proof but considering a backup will probably take about a third of the time this seems a better option.

 

EDIT: Yes the connection would be single gigabit. The fileserver is mainly used for random access of smallish Microsoft office documents. Good point about the bottleneck though, the backup will certainly be effected.

Edited October 16, 2012 by zag

[synaesthesia]

A trick which is increasingly popular with storage arrays is SSD caching - arrays of mechanical drives providing a hell of a lot of space and a relatively small consumer grade SSD handles cache. This makes a massive improvement to performance in real life terms and the SSD is literally nothing special - the most popular one used by the likes of HP and Dell are Samsung 830s.

[localzuk]

RAID is designed for prevention of disk failure. ie. you can lose a disk and your server carries on. If you don't use any form of RAID on your server, you're putting an awful lot of trust in those individual drives.

 

Your backup won't be any faster really - as you're still using a 1GbE connection... And I'd assume you're not going to backup to SSD too.

 

I really wouldn't be a fan of putting all my file server data on consumer grade drives.

[zag]

RAID is designed for prevention of disk failure. ie. you can lose a disk and your server carries on. If you don't use any form of RAID on your server, you're putting an awful lot of trust in those individual drives.

 

Your backup won't be any faster really - as you're still using a 1GbE connection... And I'd assume you're not going to backup to SSD too.

 

I really wouldn't be a fan of putting all my file server data on consumer grade drives.

 

Interesting points. For the RAID thing, I really have not had good experience with it in terms of disaster recovery. Had 2 RAID 5 failures compared to 1 Disk failure that was saved with RAID 5!!. I don't get why having 3 disks is better than 1, surely there is 3 x more chance of getting a failure at some point? I can understand the advantages of uptime though.

 

The backup server will also be a 600GB SSD as well. Now I think I need to look into a 10GB network connection!

 

The other option of course is to make a parallel file server using DFS, I wonder if that is a better idea? If I can build these servers for less than £1.6k each then it might be a good option.

[localzuk]

Interesting points. For the RAID thing, I really have not had good experience with it in terms of disaster recovery. Had 2 RAID 5 failures compared to 1 Disk failure that was saved with RAID 5!!. I don't get why having 3 disks is better than 1, surely there is 3 x more chance of getting a failure at some point? I can understand the advantages of uptime though.

 

The backup server will also be a 600GB SSD as well. Now I think I need to look into a 10GB network connection!

 

The other option of course is to make a parallel file server using DFS, I wonder if that is a better idea? If I can build these servers for less than £1.6k each then it might be a good option.

 

DFS + 10GbE would likely be a good option. It'd effectively be RAID1 over Ethernet.

 

10GbE is expensive though... Very expensive. Might be cheaper, if you have the capacity, to get a couple of multi-port 1GbE cards and bond connections together and get something like 4GbE on the server instead.

[Domino]

Interesting points. For the RAID thing, I really have not had good experience with it in terms of disaster recovery. Had 2 RAID 5 failures compared to 1 Disk failure that was saved with RAID 5!!. I don't get why having 3 disks is better than 1, surely there is 3 x more chance of getting a failure at some point? I can understand the advantages of uptime though.

 

You're increasing the chance of a single disk failure by 3, but as the array is resilient to a single drive failure you're decreasing the chance of losing the data by a third.

 

DFSR on two standard servers is a *much* better idea

[twin--turbo]

Interesting points. For the RAID thing, I really have not had good experience with it in terms of disaster recovery. Had 2 RAID 5 failures compared to 1 Disk failure that was saved with RAID 5!!. I don't get why having 3 disks is better than 1, surely there is 3 x more chance of getting a failure at some point? I can understand the advantages of uptime though.

 

The backup server will also be a 600GB SSD as well. Now I think I need to look into a 10GB network connection!

 

The other option of course is to make a parallel file server using DFS, I wonder if that is a better idea? If I can build these servers for less than £1.6k each then it might be a good option.

 

 

I would never run a server without raid,

 

quite simply if a disk fails then the server continues to run until the disk rebuilds into the raid or a new disk is rebuilt. Compare that to having to rebuild the server from scratch.

 

Rob

[localzuk]

I would never run a server without raid,

 

quite simply if a disk fails then the server continues to run until the disk rebuilds into the raid or a new disk is rebuilt. Compare that to having to rebuild the server from scratch.

 

Rob

 

But in the world of virtual servers and 'whole server' image copying for backup, rebuilding wouldn't need to happen - all that would need to happen is a copy of the backed up image to new hardware and bingo bongo, you're up and running again.

[Willott]

Check how much data you write to your current file server on a daily basis (not sure how you'd do this) - this would give you a clue as to estimated life of drive (Intel publish the write endurance of their SSDs). To look to extend the life, if you can afford to, overprovision the drives, this gives more space for garbage collection and wear levelling, less storage space, but reportedly longer lifespan.

 

I would think about having 2 servers with a single SSD in each, running DFS, then, for your backup server, have spinning rust - your backups should send data in large chunks, rather than small random accesses which your file server will have to deal with - throughput to spinning rust isn't too bad, and if you have 4x drives in RAID10 you ensure your backup data is secure, get speed benefits of RAID0 and you should have more space to be able to have a longer backup retention.

 

Of course, if one of your DFS servers goes down, get it back up as a priority, or, have 3 DFS servers (possibly one in a separate building - though you need to ensure you have decent link speed) and then having one down is less of an issue.

 

From the size of the drives and as they're being used as for fileserver only, I don't think 10GbE would be of much benefit (how much of your current connections do you currently use)? If you were looking to have the servers as SAN for Hyper-V (Server 2012 with it's hyper-v using SMB - if I've read that correctly on the brief looks I've had at Server 2012 Hyper-V), then 10GbE may be beneficial.

[Domino]

But in the world of virtual servers and 'whole server' image copying for backup, rebuilding wouldn't need to happen - all that would need to happen is a copy of the backed up image to new hardware and bingo bongo, you're up and running again.

 

But, in the real world, you can't virtualise *every* machine

 

..and you'd still be running RAID on your shared vm storage, otherwise you'd be restoring backups fairly often....

[gshaw]

Personally I'd only use enterprise grade kit for enterprise applications but maybe I'm just cautious. I remember seeing an EMC presentation where they compared standard SSDs to storage-grade ones and they were showing how consumer drives wear out much quicker in high read\write environments (although you'd expect them to say that!). Think it was to do with consumer drives writing in two directions and enterprise only uses one to save wear on the flash (although memory is very hazy on this)

 

No RAID on a file server seems very risky, although you can restore from backup would your users tolerate the downtime of losing a 600GB file server VM because one of the drives failed and then takes however long to bring it back up? Seems like you're sacrificing one of your layers of data protection for speed, is it worth the gamble? Saying that if this is a physical machine it'll take even longer to restore so it's an even bigger gamble.

[Willott]

Personally I'd only use enterprise grade kit for enterprise applications but maybe I'm just cautious. I remember seeing an EMC presentation where they compared standard SSDs to storage-grade ones and they were showing how consumer drives wear out much quicker in high read\write environments (although you'd expect them to say that!). Think it was to do with consumer drives writing in two directions and enterprise only uses one to save wear on the flash (although memory is very hazy on this)

 

I think they may have been talking about MLC vs SLC (multi level cell vs single level cell) NAND, the actual memory data is stored on. The SSD Company - STEC - CellCare Technology talks about Stec's technology to extend the life of MLC (there are other manufacturers who use other techs) - but it all depends on how much data you write to the device - EMC look at a point of view of a large enterprise who write several hundred GB of data each day, rather than a school who may write 5-10 GB data per day, not much of an SSD, so as long as wear levelling works correctly, the life of the drive may be longer than the manufacturer suggests - this very much depends on the actual amount of data written. As can be seen on the Stec page, enterprise devices have a much longer working life, but I know of a large number of comercial hosting and service providers who use Intel 320s in their servers.

 

No RAID on a file server seems very risky, although you can restore from backup would your users tolerate the downtime of losing a 600GB file server VM because one of the drives failed and then takes however long to bring it back up? Seems like you're sacrificing one of your layers of data protection for speed, is it worth the gamble? Saying that if this is a physical machine it'll take even longer to restore so it's an even bigger gamble.

 

The idea of using DFSR - data replicated and accessible from multiple nodes. If a machine goes down, force remove from DFS, replace the disk, stick a new copy of windows on, add it as a DFSR replication target, add as a DFS folder target. Fairly quick and simple.

Edited October 16, 2012 by Willott

[twin--turbo]

But, in the real world, you can't virtualise *every* machine

 

..and you'd still be running RAID on your shared vm storage, otherwise you'd be restoring backups fairly often....

 

Exactly.

 

In our mixed environment we have

 

Stand alone servers

VM hosts with Local Storage

VM Hosts with shared storage and HA.

 

The last one is the least worry. But they still have raid on them.

 

Rob

[zag]

Excellent discussion chaps, really given me some great ideas.

 

I've been using the intel 320's for a while now and they are great drives. Our file server probably gets less than 5gb written to it a day so I really don't think the write lifetime will be a problem. I'm sure I read somewhere you would have to write data 24/7 for something like 7 years to reach the limit.

 

We have had our Sims server on an SSD for 2 years now and that probably has got more IO activity. No problems so far (touch wood).

 

Our environment is moving towards Hyper-V for all machines except the domain controllers.

[tmcd35]

Interesting discussion. I have to say I'm in the camp of not running a server without some form of RAID. I'd also be wary of the read/write limits on a file server. A couple of questions do come to mind (I could google the answers...)

 

1) Whats the price difference between Enterprise grade SSD's and 15k SAS drives?

2) Do Hybrid drives (standard drives with an SSD cache) work in RAID?

 

My gut instinct is that current SSD's probably couldn't compete with SAS in terms of price/performance?

[localzuk]

The price difference is a *lot*. Intel 320 600GB is about £500. A HP enterprise 600GB is about £2,000.

[gshaw]

The price difference is a *lot*. Intel 320 600GB is about £500. A HP enterprise 600GB is about £2,000.

 

Aren't the HP Enterprise drives apparently really slow for the price as well, sure I heard that somewhere recently

 

DFS is one of those technologies that sounds wonderful but when I read threads where it's clogged up with a 20GB backlog it doesn't inspire confidence, maybe that's just the unlucky ones?