Storage Box

[localzuk]

So, I am now in the process of virtualising our network to XenServer. One of the things I want to do is to set up a nice box with lots of disks and lots of network ports, for the VMs to have their disks stored on. I plan to do this with FreeNAS or similar.

 

Any tips on makes of equipment? I don't think we're going to be able to afford to go with brand named iSCSI arrays etc... So, I'm thinking more along the lines of a couple of cheap boxes to do it instead? (One to store the images, one to store backups of the images).

 

Oh, and preferably I'd like them to be 1U rack units. But this isn't a necessity, as I can get some plugs installed near a shelf if needed.

[Ric_]

@localzuk: Sun Storage 7110 - cheap as chips on matching grant.

[localzuk]

@localzuk: Sun Storage 7110 - cheap as chips on matching grant.

 

No it isn't. It is £8600. Ok, you get 2 of them, but £8600!!

[Ric_]

No it isn't. It is £8600. Ok, you get 2 of them, but £8600!!

 

You'll have to give us an idea of what you consider to be cheap then! £4300 for 1.4TB of usable space is pretty cheap!

[localzuk]

You'll have to give us an idea of what you consider to be cheap then! £4300 for 1.4TB of usable space is pretty cheap!

 

Half that would be cheap. I don't need 1.4TB of space. I'd need maybe half that.

[teejay]

If you are using FreeNAS then any standard server from the like of HP or Dell would be fine.

Now I'm a great fan of open source products, but even though FreeNAS is a great product I would be wary using it as an iSCSI target for critical servers just purely down to the lack of guaranteed support.

[localzuk]

If you are using FreeNAS then any standard server from the like of HP or Dell would be fine.

Now I'm a great fan of open source products, but even though FreeNAS is a great product I would be wary using it as an iSCSI target for critical servers just purely down to the lack of guaranteed support.

 

In what way do you mean guaranteed support? I'm using XenServer - which is free... So no support for that. And paid for != support. There are plenty of companies that provide paid for support if needed for O/S software...

[Duke]

I was going to recommend the Sun stuff too, but if you're on that tight a budget then that's out of the question. Have you tried the Dell stuff? That's the lower-end hardware but at least there's a few options. We've got a lot of Dell hardware here and no complaints - solid boxes and good support.

 

Cheers,

Chris

[Guest Guest]

Openfiler, DRBD & iSCSI target. Enjoy.

 

just in the testing stage of our irtual infrastructure. Got DRBD working, just waiting for the 3.4TB of space to sync - oh joy.

 

FYI we have gone with 2x Dell 2950s with 4.5TB. And 2x R200s with 8GB ram and x3360 xeons (iirc)

 

 

http://www.the-mesh.org/tiki-index.php?page=OpenFilerHaSetup#Configure_DRBD

 

http://wiki.hyber.dk/doku.php/openfiler_2.2_ha-cluster_guide

[localzuk]

Ok, so I think I have my plan now. I have 2 Fujitsu-Siemens RX-220's that will soon be not in use due to being virtualised. They currently have 160GB hdd's which isn't really enough. So, my plan is to buy 4 1TB SATA disks - 2 for each box, and a quad port network card for each.

 

So that will give me 1TB of space, backed up with a second machine. Running in at around £900.

 

Now, *that's* what I call cheap. lol

[Butuz]

Now, *that's* what I call cheap. lol

 

Yes - but will it provide fast reliable storage for your Xen boxes and all of your virtual servers for the next 5 years, lol.

 

That is the BIG question.

 

If your consolidating many servers and many different storage devices into few servers and one or two storage devices - it's wise not to skimp. Virtualisation is not something that should be done on the cheap - if your storage goes down - all your servers go down, if your storage is not fast enough, all your servers will underperform. Be a bit careful.

 

If you can't afford the excellent Sun 7110 then look at HP 200i MSA. No where near as fast, reliable, or scalable as the SUN - but not as expensive either.

 

Butuz

[teejay]

In what way do you mean guaranteed support? I'm using XenServer - which is free... So no support for that. And paid for != support. There are plenty of companies that provide paid for support if needed for O/S software...

 

XenServer is supported by Citrix, just need to pay for support. From what I understand, FreeNAS isn't supported by any commercial support provider, all you have is their forums.

 

How many servers are you looking at virtualising and how critical are they?

[DMcCoy]

You'll have to give us an idea of what you consider to be cheap then! £4300 for 1.4TB of usable space is pretty cheap!

 

I did get 3TB usable on 15k rpm SAS drives for £7k with the HP 2012 (RAID6).

 

Without matching grant the prices are somewhat high. Nice features though.

 

Just about to order some Dell R710s to replace my old Xeon ESX blades. They are a bargain in comparison to non grant prices

 

 

After using openfiler for a few weeks, I'd say it's OK, not great though. The more drives the better if you want to increase IOPS.

[DMcCoy]

Ok, so I think I have my plan now. I have 2 Fujitsu-Siemens RX-220's that will soon be not in use due to being virtualised. They currently have 160GB hdd's which isn't really enough. So, my plan is to buy 4 1TB SATA disks - 2 for each box, and a quad port network card for each.

 

So that will give me 1TB of space, backed up with a second machine. Running in at around £900.

 

Now, *that's* what I call cheap. lol

 

You need more spindles if you want to host several VM images, I think you'll find a mirror to be too limiting when you have several running. I'd start looking at 6-8 disks or up.

[rklorese]

I'm using XenServer - which is free... So no support for that. And paid for != support.

 

BTW, to be a completist -- you CAN buy support for free XenServer from Citrix.

[localzuk]

Ok, so, in that case, virtualisation is well outside our budget for another 2 years. *sigh* I'll give up in that case

[teejay]

Don't give up, there is always a way round it! Do you already have the servers you were going to use as virtualisation hosts? If not, what about speccing up a server with direct attached storage which will be cheaper. Yes you lose the live motion side of it but that's not really important on small scale Virtualisation projects.

Can you give us an idea of how many servers you are looking to virtualise and what kit you already have that you are thinking of re-using and what your total budget would be?

[DMcCoy]

Ok, so, in that case, virtualisation is well outside our budget for another 2 years. *sigh* I'll give up in that case

 

You can run VMs on a mirror, I do at home, but you need to keep expectations of throughput somewhere sensible. It will be ok for sustained read, but even with a fibre array with 10 drives full random read barely breaks into single MB/s digits. If it's consolidating servers with low I/O then it will probably be fine, large numbers of disk images and snapshots will be where issues start to become apparent. File servers do not vitualise well, especially with Mac clients, although that said both of mine are virtual (having split the Mac files to another VM yesterday to reduce the load).

[localzuk]

We have roughly 10 servers at the moment. They do:

 

AD/DNS/DHCP

DNS/FILE SERVER (This will not be virtualised, as it would be silly to do so)

ISA

SQL SERVER/SIMS

Terminal Servers (32bit) (3 of) - this is a grey area, not decided whether or not to virtualise or not

IAS

Website

Internal Intranet

Ghost/Jabber/other little things

 

These are currently running on 7 HP DL360G5's (4 of which are dual quad core 2.8Ghz, 3 are dual dual core xeon 2.4Ghz - 3 have 16GB RAM, the rest have 8GB - none are utilising it all yet, as they were updated when we had some spare cash), all with dual 146GB SAS disks. We also have the 2 F-S servers mentioned earlier (dual dual core 2.2Ghz opterons, 2GB ram each, and dual 160GB SATA disks).

 

Some of the tasks above (website, intranet, IAS, main AD) are all virtualised on a VMWare box with 8GB RAM already - this has been like this for about a year and a half.

 

We will not be buying any more servers for 2 years.

[Duke]

Ok, so, in that case, virtualisation is well outside our budget for another 2 years. *sigh* I'll give up in that case

 

Virtualisation still isn't really cheap if you want to do it all properly. There's nothing wrong with what you were proposing, but it depends on what kind of SLA you want to have. One of the main reasons I'm looking at virtualisation is to improve our level of failover and redundancy. With a mirrored SAN and Fault Tolerance from VMware I know I could theoretically get close to zero unplanned downtime. I've had quotes to do less than that which ranged from £20k to £120k!

 

I'm sure the Fujitsu boxes would do the job, but Butuz is right - will it be a long term solution that will support the growth, expansion, performance and uptime you need? If you've only got a couple of £k to work with then you'll struggle to get a redundant solution, but it's plenty for proof of concept and development if you need to persuade anyone higher up that it's a worthy project.

 

I did a proof of concept using two Dell PCs running Xen (the trial that gives you HA) and our NetApp SAN (I could just as easily have used the Sun S7000 Simulator) and went from a potential downtime of days for a complete critical server failure to it coming back up in 1m45s. If you need to persuade a management team on this, tell them it's not whether they can afford to do it, it's whether they can afford not to...

 

Cheers,

Chris

[localzuk]

As I've said - 2 years is our timeframe for buying any new servers. So, in 2 years time we'd be looking at a budget of about £15k in total for new servers...

[teejay]

We have roughly 10 servers at the moment. They do:

 

AD/DNS/DHCP

DNS/FILE SERVER (This will not be virtualised, as it would be silly to do so)

ISA

SQL SERVER/SIMS

Terminal Servers (32bit) (3 of) - this is a grey area, not decided whether or not to virtualise or not

IAS

Website

Internal Intranet

Ghost/Jabber/other little things

 

These are currently running on 7 HP DL360G5's (4 of which are dual quad core 2.8Ghz, 3 are dual dual core xeon 2.4Ghz - 3 have 16GB RAM, the rest have 8GB - none are utilising it all yet, as they were updated when we had some spare cash), all with dual 146GB SAS disks. We also have the 2 F-S servers mentioned earlier (dual dual core 2.2Ghz opterons, 2GB ram each, and dual 160GB SATA disks).

 

Some of the tasks above (website, intranet, IAS, main AD) are all virtualised on a VMWare box with 8GB RAM already - this has been like this for about a year and a half.

 

We will not be buying any more servers for 2 years.

 

Right,

Terminal Servers, SQL Servers and SIMS it's not recommended to virtualise them. However, what we're doing is virtualising them as single servers on there own machines to aid with DR, as you can then just take a backup of the VM files.

All your web stuff could be virtualised onto one of your existing servers no problem. Maybe get some extra hard drives for a couple of your DL360's and start moving stuff across onto those if you need extra storage space for the images and see how it goes.

Really wouldn't bother faffing about with iSCSI unless you can afford the decent boxes. You've got good enough kit there to virtualise quite a bit with your existing kit.

[Duke]

Terminal Servers, SQL Servers and SIMS it's not recommended to virtualise them.

 

For what it's worth, I believe Capita are now supporting SIMS in a virtual environment, and the old adage of not virtualising SQL and Exchange apparently doesn't apply too much any more as long as you've got hardware virtualisation support on the CPUs and decent IOPS from your storage. Just a thought if it's going to free up any more hardware for you to play with.

 

Cheers,

Chris

[Butuz]

Ah I can see where your problem is now - you've already invested heavily in server infrastructure so it becomes more difficult for you to present your case.

 

My situation is the same as yours nearly - 10 servers all doing separate things (inc 3 terminal servers). Where I have extra leverage is because 5 of the servers are older than the internet itself - out of warranty (thus are costing us ££ per year to warrant), and under perform. In my case - 10 servers virtualised down to 4 physical servers (2 new, 2 existing) with a 7110 san is going to provide big increases in performance, reliability, redundancy - and big reductions in yearly warranty costs and electricity. It'sa win win. there is no reason why not to (apart from initial cost, speculate to accumulate).

 

From what I can see - benefits to you in virtualisation are going to come more from reliability, redundancy, increased uptime, etc rather than performance. I think you also have room to consolidate and get rid of some of the older non quad core servers, thus saving some electricity and support costs.

 

You have to present the benefits to SMT - fault tolerance is the main one, currently if a physical server goes down, all services on it are down until it can be repaired (few hours, through to over a week) - is this acceptable? To us it was not. With virtualisation, if a physical server dies, so what - another physical server will take the strain. ifnact two could fail at teh same time and it still wouldnt be much of a problem No downtime apart from 2 or 3 minutes while the VM reboots. This is a key benefit with massive inpact for all teaching learning and administration, and one that if your SMT understand, probably will want to look at finding extra money for a decent SAN.

 

The worst thing you could do would be to implement an poor/unreliable/non redundant SAN - and then have it all go pear shaped - you will have a seriously hard time convincing SMT that virtualisation is a good idea when all your VMs are down

 

As for what to virtualise - yes you can virtualise SIMS, yes you can virtualise terminal servers, yes you can virtualise DC's. you just have to ensure that you have the approprate hardware and storage system specified so that you do not take any performance hit.

 

Butuz

Edited May 12, 2009 by Butuz

[teejay]

@Butuz

One question, what happens if your 7110 goes down? Unlikely, but it is a single point of failure and would have a bigger impact than one server failing.

 

What I was trying to get across before is that there are some advantages to doing virtualisation within his current infrstructure, as there are still benefits when just running with direct attached storage as you have the abstraction layer between the hardware and the O/S so that machines can easily be moved. A lot of his web infrastructure can happiliy sit on one of his existing servers no problem, especially the linux based stuff. A server could then be kept with up to date images to bring online if required due to a failure.