SAN kit purchased, no turning back!

[MrWu]

Exciting times...and thanks to all the guys here for patiently helping me out with suggestions.

 

Having used to hosting Hyper-V on local storage for years I have, after much procrastination (and frustrations with no solution fitted the need) purchased an HP MSA P2000 G3 SAN (on a 50 percent cashback and a good deal even without that - Cheers Luke@Millgate!)

 

I choose the 6GB SAS version because:

 

1) Good compromise with cost against FC and 10GBe

2) Should have better performance than vanilla iSCSI

3) Don't envisage the school using more than 4 hosts, so I can have multipath direct to the SAN without mucking about with redundant 2xFC switches, 2x10Gbe switches etc, bit simpler.

4) Cost of HBA cards hell of a lot cheaper than 10Gbe NICs or FC HBA cards.

 

I have a DL380 G7 already running Hyper-V (dual Xeon 5620,24gb RAM, 8 x 300gb on Raid 10) running Finance SQL package,WSUS and Impero, hardly sweating at the moment

 

So will purchase another 2 x DL380 G7 (also Dual Xeon 5620, 2x146Gb 15k, 64Gb RAM) as hosts, plus 10Gbe NIC for VM to LAN communication.

 

Will bring the existing DL 380 G7 to spec as well, then transfer the VMs to SANs. With some RAM carried over I should have in the end 3 fairly well specified Hyper-V hosts with 72Gb RAM each. I'm also looking at using a another HP located in another block running a Virtual DC.. replicating all VMs with Veeam as a DR solution (not as comprehensive as a dual SAN set up, but again cost was a factor)

 

Have a few questions if you guys don't mind:

 

I'm planning to run SIMS, Exchange, Finance SQL Package (Correro), WSUS, Intranet, Impero, RDS - I hope to move the more mission critical stuff in the list once the MSA P2000 G3 has proved reliable say October. I am planning to buy 10 x 600GB 10K SAS, and maybe using the 6 x 300GB 10K SAS (that would be surplus from the existing DL380 G7) in there as well.

 

So with 10 x 600GB 10K SAS, 6 x 300GB 10K SAS, how would you curve up the RAID for different access needs? Would you not bother with the existing 300Gb SAS but get bigger/faster SAS drives instead? (again I did it for cost reasons and reusing old components)

 

With 10Gbe Ethernet for VM to LAN, Would you just bung all your VMs onto that one pipe for bandwidth or would you use Virtual LAN software to curve up the 10Gbe pipe(or again keep it simple?)

 

Looking forward to setting this up ! :-)

 

 

Cheers all !

[SYNACK]

We have a similar setup, one pol of fast sas drives for active data in raid 6 and a stack of sas drives for a backup staging area and archive. The sas interface is really good and faster by far than iscsi unless you're doing 10gb.

[MrWu]

Thanks Synack,

 

That is reassuring, I was never sure vanilla iSCSI would have carried the school for 3-4 years if I implemented it now. 10Gbe / FC was coming up £23k-£40k for a decent, redundant setup(two switches, one SAN) .The SAS solution, with a bit of poking about with qouting, is coming under £15k. leaves me enough to get some SSD drives into teachers PCs as a bonus!

[SYNACK]

Yea, we went iSCSI at one place around 4-5 years ago and when it came to doing the next place and SAS was avalible it was a nobrainer, cut 20k off the price add to the speed and cut out the need for a whole bunch of extra switches. For our deployment it would nave been stupid not to.

[speckytecky]

Interesting post MrWu, I take it you and SYNACK are running pretty big establishments - with pretty hefty budgets. Makes the set up I inherited sound like peanuts - it was a single Windows 2003 Server and circa 80 clients on XP with SIMS running in the Admin office also off an XP machine in 2007; now expanded to a 2008 standard curriculum tower with a 2008 R2 as second DC and SIMS server, mix of Win 7 and XP clients 130 all told, about a dozen MACs and Mini Mac Server. I suppose now I need to start looking toward exploring the ways forward into the next few years?

[MrWu]

Hi speckytecky;

 

The school I work in has 1100 students and 150 staff. Coming from a Small corporate background it was gigantic lol. The network was very tired though, servers were nearly on it's last legs (one CCTV server is over 10 years old, still going though), no proper UPS systems, cabling all over the place, AntiVirus system wasn't implemented properly, no documentation, 6 months down the line, we have a Windows 2008 Domain (soon to be native) Forefront 2010, APC UPS protecting the servers (soon to have a Borri 10kva hopefully powering the whole server room as phase 2 in summer, UniFi Wireless (school wide in Summer) and this Hyper-V project. A lot of the money came about when disasters happened since I started there (I'm a bad luck charm!) Powercuts, hardware failures, network failures..so convinced the SMT that investing in a good network would stop fire fighting and ICT could actually start proactively developing the school's future in ICT.

 

I suppose my advice is to talk to good folks here, but focus on piorities first but take your time phasing your project. My mistake being a new boy is agreeing to the timescale of getting this done,so ending up with a hefty summer workload.

Get your network infrastruture right first (again, blinkered into replacing old servers and not auditing the cable infrastructure thoroughly I underbudgeted and had to fight for more money to get the school recabled with OM3 fibre - a long term investment worthwhile.)

 

Virtualising is very popular but again the thing that did my head in most, so much options and cost implications. you will get solutions thrown at you at £50k plus that claim you could saw it in half and still works, plus saving loads of electricity...my argument is do a school need millisecond switchover and whats the saving in power vs me frisking another £20k from my school when they desperately need PCs and projectors replacing, after all, teachers don't thank you for buying a £13k Chassis switch.

 

So from my point of view, with the money I was given, how can I make maximum use of that money (in my case good backups, a realistic SLA ..quick rebuild, replicating to a server off site to run important VMs if SAN or server room burns down etc)

 

Yes and some of the schools I have seen that could easily spend money on a core switch that would consume the money I set aside to replace all my edge and core, not saying that's wrong, those schools might have established different requirements.

 

and No matter how small your network you inherited is, it's your baby and be proud of it. :-)

Edited July 1, 2012 by MrWu

[SYNACK]

Our places are 500-600 students each with 200+ stations. These run stacks of software centralised, remoteapps for the dirty SMS and local email, filtering, AD, AV, Papercut, local web sites etc. There is also video shareing with clickview etc. These networks have grown to a reasonable size to remove as many limits as possible on stuff like storage space and speed.

 

While this does mean a decent amount of investment it means we have a fast robust network that can have new services chucked in without much hassel. It also means that the stuff lasts for a rather long time if properly maintained as it supports the standards of the time and has overhead and upgrade potential to keep it going well in future.

[mavhc]

Is it worth buying 10k/15k 2.5" hdds these days, compared to the prices of ssds?

[SYNACK]

Is it worth buying 10k/15k 2.5" hdds these days, compared to the prices of ssds?

 

SSDs do everything faster, including die (when they go they loose everything instantly) hence the reluctance to use it as a primary storage medium for the time being. Also there are still very few large RAID devices designed for SSDs and they will actually make it slower (than it could be, faster than SAS though) given the various levels of cache and command optimisation which actually add extra latency so without one designed for SSDs you are not going to get the best performance out of them anyway. If I was using SSDs on a SAN I'd be looking at least RAID 6 preferably 1+5 to double the redundancy and give you a valid chance of avoiding at catastrophy.

Edited July 1, 2012 by SYNACK

[MrWu]

SSDs do everything faster, including die (when they go they loose everything instantly) hence the reluctance to use it as a primary storage medium for the time being. Also there are still very few large RAID devices designed for SSDs and they will actually make it slower (than it could be, faster than SAS though) given the various levels of cache and command optimisation which actually add extra latency so without one designed for SSDs you are not going to get the best performance out of them anyway. If I was using SSDs on a SAN I'd be looking at least RAID 6 preferably 1+5 to double the redundancy and give you a valid chance of avoiding at catastrophy.

 

Yeah, was thinking of using SSD drives for my Remote Desktop Servers (they are pretty easy to reimage) The HP stuff is well expensive (a 100Gb to fit a DL360 G7 cheapest is £400 each!)

[mavhc]

Buying SSDs from computer makers is some sort of joke/scam, they'll charge last year's prices for last year's models, and then add 50%

[alttab]

Is it worth buying 10k/15k 2.5" hdds these days, compared to the prices of ssds?

 

generally speaking, SSD storage on entry-level and midrange arrays look to be affordable only in specific use case of SSD's as an option

to improve on cache read-write speed. If the cost of engineering FC drives for entire disk shelves is seeing a wholesale move to SAS, due to the

reduced cost of SAS backplane connectivity and ability to intermix SAs and SATA as is the case with kit like P2000, it's obvious which one is

the most cost affective for the foreseeable for entire disk shelfs on the entry level out of SAS or SSD. i think SSD is still niche for a few more years

 

For Server internal storage, cost and reliability is a big driving force. If you want to go down the route of stuffing 2U servers full of disk rather than

going for 2.5inch drives in an exrternal SAS or iSCSI array then from a price-performance-capacity is there anything on the SSD side that can beat a server chock full of 600gb SAS ent disks ?

[jamesfed]

Yeah, was thinking of using SSD drives for my Remote Desktop Servers (they are pretty easy to reimage) The HP stuff is well expensive (a 100Gb to fit a DL360 G7 cheapest is £400 each!)

 

We are working on something simlar this summer accualy - a farm of 4 RDS servers with OCZ Vertex 4 drives in

 

Will post up the results once we get everything in place.

[MrWu]

We are working on something simlar this summer accualy - a farm of 4 RDS servers with OCZ Vertex 4 drives in

 

Will post up the results once we get everything in place.

 

Cool! Are you using PCIe stuff? shame the HPs only let you us thier own brand HDDs on the servers.

[jamesfed]

Cool! Are you using PCIe stuff? shame the HPs only let you us thier own brand HDDs on the servers.

 

Not this time round (have PCI-E SSD in all of our Hyper-V servers now though), biggest reason not to was we will be using custom 2U cases and half height PCI-E SSD isn't cheap so just normal 2.5" drives this time round.

[MrWu]

Got it, suppose being remote desktop servers there's probably not a need to go for belts and braces HPs, since at less cost you could build a redundant RD farm anyway.

 

On a complete tangent, just read your blog on VDI and HP's Flash write back cache

 

http://www.serversexpress.com/hp-534562b21.html

 

Is it good performance boost for Hyper-V stuff too? So the module replaces the 256mb card in the server, without the need for the battery module unit?

Edited July 1, 2012 by MrWu

[jamesfed]

Yep not that I wouldn't use propper HP kit if I had the money at £1,400 for the farm of 4 though its a easy call!

 

Using cache always depends on the work load if its a lot of random read/write then you should almost always see a boost but if its easy sequential stuff then you wouldn't see as much of a boost.

 

The flash kit uses super capaciators to keep things powered more details here on page 10 - http://h20000.www2.hp.com/bc/docs/support/SupportManual/c00687518/c00687518.pdf

[Arthur]

Yep not that I wouldn't use proper HP kit if I had the money

I probably wouldn't choose HP SSDs though...

 

Benchmarking a $11,699 HP Enterprise Performance SSD

 

^ Cache on the RAID controller definitely helps.

[jamesfed]

No way! for $11,699 I would be expecting one of the Fusion-IO cards - those things FLY.

 

Anywhos all off topic back to the SAN

[MrWu]

Shiney new MSA P2000 G3 sitting in my office, lovely and heavy bit of kit. Now to order the rest for summer hols. Will keep you posted!

[MrWu]

Ok i thought I may as well keep this thread going with thoughts and ideas plus reasoning, maybe not be always the right choice, but hopefully someone can read and chip in with suggestions or share experiences.

 

Well with the summer looming and the fact that I was not CCed in the email about purchasing deadlines(which was last week!) i had to speed up my purchase somewhat!

 

HP by the way have extended their 50 percent cashback on the MSA P3000 SANs and free second processors for selected DL G7 and G8 until July 31st (you need to buy double the memory to qualify-but vs free Xeon you are still quids in) ..you can have up to 5 servers on one PO, which is pretty good if you start with your 3 host and SAN purchase.

 

So considered virtualising fileservers onto the SAN but eyeing the future with Hyper-V r3 with SMB replication(and the fact that budget,if not used will be spent on a Statue outside school!) will go for 2 x DL380 G7 with 2 x 146gb for OS and 6 x 1Tb for storage (mmm Raid 5 or 50?) thinking outside the box (or my backside) i may run Hyper-V on these 2 HPs with VM fileservers. Also use Veeam to replicate important VMs from the SAN to have standby for now.

 

Well been busy at school so thinking up this stuff while sorting out movie maker and resetting students passwords, so sadly my SAN is still sitting in it's box.

[MrWu]

Thinking about performance and load on the VMs and Hosts today, with 3 hosts (each with 2 x Xeon 5620,72Gb RAM)

 

Some have been running 10 - 12 VMs on something similar to this spec, how do you assign virtual cores in conjunction with machines, at present I would normally not over assign virtual cores vs actual cores on the host.

[DrPerceptron]

Disk performance is usually the performance limiter...

 

I was told that a single physical core is usually good for 6-8 virtual cpus. But obviously what you're making those virtual cpus do all day will also determine how you dish it out.

 

The majority of our virtual servers are at least 2 core and we're running around 10 per host.

[MrWu]

Indeed. I will be having 12 x 600gb 10k SAS, will draw up a little plan later on tonight to curve up the RAID.

So choice is :

 

Have all 12 Raided over lots of Spindles, but risk one application swamping the IOPS(however our total servers at present has extremely kow IOPS after running perfmon.)

 

Have the 12 HDDs divided into seperate RAIDS according to SQL / Exchange etc..but risk of over complication.

 

I also have at least 6 300gb 10k SAS I could shove in the P2000 as well.

[SYNACK]

@MrWu Many spindles make light work, I'd chuck the whole lot in a RAID6 pool possibly with a hot spare. Divide it up using LUNs and partitions as this will allow the storage to be more quickly reconfigured/divided up and also allow for better utilisation of the available resources. I don't see the spindles all being busy thanks to one app. I'd look at chucking the smaller drives into another array of RAID6 which you can use if required for more storage and even to split the load in the unlikely event that your usage can saturate a 12 spindle SAS set.