Server spec out for potential virtualisation project

[localzuk]

So, I'm plotting for the future again and as part of our possible move to being an academy, we will end up taking on a bunch of server roles that the LEA currently provides, meaning our current set up isn't up to the job of it all.

 

Rather than faff around with adding more servers to what we have at the moment, and therefore chewing through more electricity I am thinking of putting everything onto a proper virtualised system.

 

For this, I am thinking of the following bits of kit:

 

2 x HP DL360 G7 servers with 36GB RAM, 2 x SATA hard disks (cheap and cheerful) - it has 4 1GbE connections also.

2 x HP DL370 G7 servers with 36GB RAM, 2 x 146GB 15k RPM SAS hard disks, 2 x FirePro V5800 graphics cards, and 4 1GbE connections (for terminal servers, with RemoteFX)

1 x Overland Storage S1000 with 12 x 300GB SAS drives, and with 4 x 10GbE SFP+ ports

 

What do people think about that sort of spec? It'd be running everything - so, file server, AD, database server, print server, various web based apps (eg Oliver), Exchange email, Moodle and whatever else I throw at it.

 

The 2 DL370's would be dedicated to Remote Desktop Services.

 

Thoughts?

[irsprint84]

I m a biggy for virtualisation and even though I have my sql virtualised I always advise physical SQL and if you re going to virtualise SQL dont HA mode it! this should be similar to my setup but I m having cheapy server separate from SAN on own box

 

What quotes did you get?

[localzuk]

Not got quotes yet. Just looking at spec.

[pete]

It seems a bit light on storage capacity given (afaict) that Overland unit doesn't have dedupe options.

 

This assumes you're slinging user data on there as well as vms. If you're creating luns with different raid setups, your storage gets chopped down even further.

[Soulfish]

I'd also say that the 10gbe on the overland is almost certainly overkill. I'm not sure you'd get past saturating a couple of GigE in most setups. The other thing it may be worth doing is holding back for the dedicated RemoteFX addin cards. Should provide better performance than high end graphics cards for RemoteFX

[localzuk]

It seems a bit light on storage capacity given (afaict) that Overland unit doesn't have dedupe options.

 

This assumes you're slinging user data on there as well as vms. If you're creating luns with different raid setups, your storage gets chopped down even further.

 

The storage system can expand as and when needed. That'd give us a couple of TB space, which is way more than we use or need. (We use about 600GB at the moment in total for user data, and virtualised servers simply don't eat lots of space). Our network is simply not going to expand so fast that we can't plan and add an extra expansion unit to it.

 

I'd also say that the 10gbe on the overland is almost certainly overkill. I'm not sure you'd get past saturating a couple of GigE in most setups. The other thing it may be worth doing is holding back for the dedicated RemoteFX addin cards. Should provide better performance than high end graphics cards for RemoteFX

 

This would likely be needed by July, are those dedicated cards going to be out by then? The 10GbE bit I can see your point, I wonder what the difference in pricing is between the 1GbE units compared with the 10GbE.

[DMcCoy]

I'd also say that the 10gbe on the overland is almost certainly overkill. I'm not sure you'd get past saturating a couple of GigE in most setups. The other thing it may be worth doing is holding back for the dedicated RemoteFX addin cards. Should provide better performance than high end graphics cards for RemoteFX

 

Although not with multiple hosts with multiple Gb cards using the same SAN. The problem with multiple Gb cards in the SAN is that teaming isn't particularly great, nor is MPIO. I put 10Gb in our SAN this time round as I already have 4 hosts connected to it with two dedicated Gb cards each, throw in backup too and the extra bandwidth can be useful. Can get expensive depending on the 10Gb connection though, along with not very bendy cables when using copper 10Gb!

[localzuk]

So, any other thoughts? The 2 terminal servers would be used by around 100 clients. Should I be looking at more of them, or more RAM? Or maybe splitting them in 2, with 2 sets of hard drives, with virtual servers running one on each set of disks?

[dhicks]

1 x Overland Storage S1000 with 12 x 300GB SAS drives, and with 4 x 10GbE SFP+ ports

 

I'd skip having any central storage for VM harddrive images and use your VM servers' local drives instead, then you don't worry about having to connect up the VM server and your central storage device. You'd still need a decent-sized storage server, of course, but I'd simply build one - Antec do rack-mount cases if you require rack mount equipment, and you should be able to fit a good number of 2 or 3TB disks in those. A decent (£1,000-ish) RAID card should provide decent performance instead of using SAS drives.

[Ric_]

@dhicks: I'd disagree about the cenralised storage unless you have some way of mirroring the VM storage so you have a copy on each. This would give you higher availability if not live migration or HA.

 

Personally, I would look to getting two identical storage servers and replicating the data so that you can eliminate a single point of failure. The 10GbE NICs are overkill too because you'll never shift the data quick enough off the disks on that box - the price of a suitable switch (of which you ideally want two) is going to be similar to that of the storage and you haven't started to factor in 10GbE NICs for your servers.

[dhicks]

I'd disagree about the cenralised storage unless you have some way of mirroring the VM storage so you have a copy on each. This would give you higher availability if not live migration or HA.

 

I'd use DRBD, which mirrors block devices over the network.

 

I would look to getting two identical storage servers and replicating the data so that you can eliminate a single point of failure.

 

For storage of user files, I'd go for one 10*2TB disk file server with a hardware RAID card and one 10*3TB disk backup server running software RAID. Hardware failure (other than individual harddrives, which would be taken care of by RAID, obviously) should be very rare and the backup server could take over for the time it took you to get the main file server back up, it might just run a bit slower. This would allow you to then have a backup server that gave users a file share containing a versioned copy of their files - yesterday's, last week's, etc, so restoring an old version of a file is a simple copy-and-paste operation.

[localzuk]

See, what you're suggesting flies in the face of pretty much all advice I've ever seen about virtualisation projects, with the idea being that nothing is restricted to a single device, so that for HA you can switch to a different host in minutes and not have worries about VMs being on one server or another.

 

The reason I specced 10GbE cards on the Overland is that a) we'd be getting 10GbE modules for our core switch anyway and b) why faff with port trunking if you can just do it in one connection?

 

Also, the reason I tend to go for devices designed for the task (eg. overland storage or similar) is that I know that company has spent a fair amount of time making sure it does what I want it to, and they know about it and can support it properly rather than relying on my learning more about the nuts and bolts and doing it myself.

[Ric_]

@localzuk: I'd agree with picking a 'proper' device. There are some extremely cheap storage solutions from Synology, Thecus and Qnap on the XenServer HCL too.

 

I'm still not convinced that you would saturate a trunked 1GbE setup. The other advantage of a trunk is you can survive a port or cable failure.

 

Following a recent storage problem, I'm putting quite a lot of thought into re-designing my storage to add more resilience. When I figure it out I'll write about it but I'm currently in the 'my head hursts' stage.

[dhicks]

See, what you're suggesting flies in the face of pretty much all advice I've ever seen about virtualisation projects, with the idea being that nothing is restricted to a single device, so that for HA you can switch to a different host in minutes and not have worries about VMs being on one server or another.

 

The VMs would exist on two servers - DRBD mirrors writes between both devices, so what's written to one disk (or RAID array, etc) gets written to the other. Network-traffic wise, you only have to send writes, reads can happen locally on each machine, and at local SATA / RAID card speeds. DRBD works very well with Xen - when used with HA monitoring tools, Xen's documentation reckons something like 0.3 seconds of downtime when switching VMs between physical machines. I think (not sure) you might need identical hardware on the two machines you are switching between to do that, and there's a minute or two (?) beforehand of set-up time while the system syncs the RAM of the VMs before it switches.

 

For some reason (probably something to do with SAN sales people...), "server virtualisation" has come to mean "bunch of processing machines with central, shared storage". However, I don't think most schools require a setup like that.

[localzuk]

@localzuk: I'd agree with picking a 'proper' device. There are some extremely cheap storage solutions from Synology, Thecus and Qnap on the XenServer HCL too.

 

Wow, the Qnap stuff is very cheap! QNAP TS-859U-RP+/8TB

[dhicks]

Wow, the Qnap stuff is very cheap! QNAP TS-859U-RP+/8TB

 

That (£2,000-ish) really isn't bad compared with most other ready-made devices. We have a QNAP server here - we initially had a bit of trouble as someone had forgotten to screw the motherboard onto the case, but it seems to be working fine now. From looking at the inside, it does seem to consist of an Atom motherboard with 8 on-board SATA ports running a BSD-based OS - you could make your own (and double the capacity) for half the price.

[john]

The HP DL360s will be great machines for you, I have them, well the G6's of them and they are excellent bits of kit and fly nicely in my network lots of happy virtual machines and nice and reliable. Each host has 4x 1g ports, one for management, one for storage access and two into the main academic network as a bonded pair and its really performing well for us which is great

[Butuz]

The spec looks fine I think it will work well.

 

A few points.

 

No experience with overland here but in my humble experience do not skimp / build your own SAN, especially if you are only going to be using one SAN (single point of failure) - buy a proven vendor one with same or next day warranty and support.

 

10GBE is a good idea on your SAN especially if you think you may expand over the next 5 years. Bonded/trunked 1GBE has overheads and caveats with virtualisation software and in really high traffic situations 10GBE will be much better. Saying that I've not come close to satuating the 3 x 1GBE I use on each of my SANS (not bonded).

 

Enjoy the benefits of virtuialisation, of which there are many!!! And it does not need to cost a fortune, infact it does not need to cost anything!!! You get 90% of the benefits of virtualisation for free with the free version of ESXi for example, the question I guess is are the other 10% of the benefits (high availability, automatic failover, etc) worth paying thousands upon thousands for. For my school (and personally I believe for most schools), the answer is no.

 

Butuz

Edited February 19, 2011 by Butuz

[jamesfed]

I'd love to hear from you if you've played around with RemoteFX - were looking at it here and have got some HP and 10zig thin clients on the way to play with (are going to be running them off my home desktop PC with its 9800GT).

You might also want to have a look at the D385 - higher core count with the new AMD 8 and 12 core opterons (potentialy 24 cores total) but only one graphics card slot as I understand.

[localzuk]

Home grade graphics cards aren't going to work according to Microsoft - RemoteFX needs a workstation class card like the ones I specced above. Also, HP says that the DL370 G7 is the only rackmount server that will work with RemoteFX and graphics cards at the moment (See here).

 

Your clients will need to support RDP 7.5, which is currently only supported by windows 7 based thin clients AFAIK (MS are releasing a new thin version of Windows 7 for assurance customers to replace their 'Windows Fundamentals for Legacy Computers' software, which should allow some thin clients to be updated to work with it).

[jamesfed]

Home grade graphics cards aren't going to work according to Microsoft - RemoteFX needs a workstation class card like the ones I specced above. Also, HP says that the DL370 G7 is the only rackmount server that will work with RemoteFX and graphics cards at the moment (See here).

 

Your clients will need to support RDP 7.5, which is currently only supported by windows 7 based thin clients AFAIK (MS are releasing a new thin version of Windows 7 for assurance customers to replace their 'Windows Fundamentals for Legacy Computers' software, which should allow some thin clients to be updated to work with it).

 

Already had it running on the 9800GT with the Beta RemoteFX works with quite a few cards with varying results but naturaly in a production enviroment itd be nVidia Tesla or FirePro.

As far as that HP doc goes we've been looking at the Workstation blade servers as our first choice.

[localzuk]

No experience with overland here but in my humble experience do not skimp / build your own SAN, especially if you are only going to be using one SAN (single point of failure) - buy a proven vendor one with same or next day warranty and support.

 

We've been using Overland for about 4 years now, our main storage device for user documents is one of their old ones (when they were part of Adaptec). So, I know their stuff is good from experience.

[m25man]

The 380 G7s are a 2u chassis but even when fully loaded with dual Quads and 36Gb they are near on silent running and only seem to have 450w psu's in them which is handy if you need to build a 3 node cluster in a cabinet that happens to live next to your desk....

 

The 1u 360's will be fine as long as you dont have to live with the constant hum of the cooling fans in the background all day.

 

We hook these up to ReadyNAS 4200's with 10Gbe on SFP+ via a GSM2328S

The RN's with 12 spindles and 10Gbe is fully certified for VMware and Hyper-V they support MPIO and CSV's

You could knock out a 3 node Hyper-V cluster with a 10Gbe SAN and still have change out of £12k (OS licenses not included) so if your on a tight budget for your first SAN project you could do a lot worse...

[dhicks]

2 x HP DL360 G7 servers with 36GB RAM, 2 x SATA hard disks (cheap and cheerful) - it has 4 1GbE connections also.

2 x HP DL370 G7 servers with 36GB RAM, 2 x 146GB 15k RPM SAS hard disks, 2 x FirePro V5800 graphics cards, and 4 1GbE connections (for terminal servers, with RemoteFX)

1 x Overland Storage S1000 with 12 x 300GB SAS drives, and with 4 x 10GbE SFP+ ports

 

Couldn't you just stick some more harddrives in those DL360s and save having to buy (and power) a separate storage server? 2TB harddrives are cheap and energy efficient, or maybe nearline SATA drives would offer the best price/capacity/performance. I assume there's a RAID card option for the DL360 (we've just bought a couple of the Dell equivilants, the R510, and I know there's a RAID card option for those). For backup, you could have one DL360 mirror and version the file storage on the other.

[localzuk]

Couldn't you just stick some more harddrives in those DL360s and save having to buy (and power) a separate storage server? 2TB harddrives are cheap and energy efficient, or maybe nearline SATA drives would offer the best price/capacity/performance. I assume there's a RAID card option for the DL360 (we've just bought a couple of the Dell equivilants, the R510, and I know there's a RAID card option for those). For backup, you could have one DL360 mirror and version the file storage on the other.

 

You don't seem to be understanding that I want a HA system with centralised storage (quite possibly I'll end up with 2 storage devices, with their built in mirroring set up). I don't want traditional 'storage' on a device, as it makes disaster recovery more difficult. I want centralised storage, per the industry standard model for virtualising servers. If I leave, I don't want to leave behind some custom built, complex to manage system. I want companies able to support their stuff, and I want it to be simple. I don't want to spend time fiddling.

 

I want manufacturer warranty support at the server level, not at a component level - ie. I don't want to have to spend time figuring out what is wrong when something breaks, I want to call HP and say 'our server broke, come fix it'.