Virtualisation and other stories

You've probably seen some of my previous posts on the virtualisation thread. Here's an update on where we are and what we are going in the virtual world. I hope it's of some use to you.

Background to recent changes at the chalk face

The move to Windows 7, Office 2010 and IE9 has been an interesting one, throwing up all manner of issues we would not have even considered possible just a few weeks ago. All seem to be resolved apart from an Impero issue with internet blocking (or not) and multiple tab opening and SIMS not working well with Office 2010 (there's a suprise). System Centre works a lot better with Windows 7 machines. We thought MAK activation might be an issue (we decided to use this in favour of KMS can't remember why) but ended up being easy. Sharepoint 2010, Exchange 2010 Lync 2010 all working nicely. In fact everything seems to be OK. Was this a good move? Yes I think so, no more OS or Office moves for another 7 or 8 years. I'll be long gone by then!

What's all this got to do with virtualisation? Quite a lot as it's the front that drives the rear (if you'll excuse the double entendre). That and the growing and ever changing needs of the staff and students. The school's reliance on school hosted services, therefore, (our decisions as other solutions we unreliable and inflexible) makes the SAN and ESX host decisions critial.

Virtualisation changes

Times are hard and money short even for academies. Two of our ESX's and the primary SAN are up for renewal in April next year after 5 years of perfect service. This time I'm going for simplicity. As the IBM kit we have is so reliable, I am moving to just 3 ESXs each with 48Gb memory, consolidating to a single SAN (rather than 2/3) and removing the SAN switch. I will, however, be using one of out older SAN switches to add our backup solution, Veeam, but the ESXs will access the SAN directly. Why are we reducing to 3 ESXs? We already run 40+ virtual servers and I cannot envisage us expanding much more over the next 5 years other that file, mail and intranet services which rely on storage capacity rather than ESX memory. I defy anyone to show me a school in the UK that needs more than this!

The choice of server is a no brainer... IBM x3650 M3. Good reliable service - worth every penny. We will be buying with 5 years warranty which, to be honest, we have only ever used once on a faulty CD drive 6 months after purchase. You can probably get faster but I'll go for the reliability every time. We expect to pay about £7,500 per unit (£1,400 pa)

The choice of SAN, though, is a little more difficult. I have been interested in the new technologies around at the moment, in particular Tintri (a demonstration of which we are going to next month) offering a 4U box with 12Tb made up of SSDs and SATA. The claimed IOPs are impressive, I only hope the price and reliability are equally impressive. Our storage requirements, unfortunately, are large and a single Tintri unit would only just be big enough for what we have now so we would have to consider buying 2 units. This, I think, will put the solution beyond our means. The replacement for our IBM DS3400 will probably be the IBM DS3512 wich has more capacity (supports 96 drives) and more cache. We will probably buy the unit with 4 expansion trays (populating only 3 of these fully with 600Gb 15K SAS drives in the first place) and turbo performance. I did look at remote mirroring but to implement that would cost in the region of £13,000 (including the hardware) not cost effective given our SLA. I expect to pay no more than £35,000 for the SAN. Which works out at around £7,000 per year, interestingly that is 4 x less than we pay per annum for our internet connection (yet another story soon to be resolved)! It's a hell of a lot more reliable and delivers a damn site more services!

The total cost of maintaining the SAN + 3 ESXs is therefore £11,400 pa. Not a huge amount in the scheme of things.

Whatever direction we take in the storage arena our rack space rather than decreasing, is actually increasing as we consolidate more of the physical kit.

40+ servers??? What on earth? Does each server run a separate role?

On Server 1 I had the following...DHCP, DNS, ADUC, Print Server, Kerboros - For Admin/Staff
On Server 2 I had the following...SQL, Exchange 2010
On Server 3 I had the following...DHCP, DNS, ADUC, Print Server, Kerboros - For Students
On Server 4 I had the following...IIS 7.5, KMS Server (for all devices), ADCS, ADLDS, Radius - For all people in the school
On Server 5 I had the following...File Services - For all people in the school
SAN 1 - File Storage
SAN 2 - Backup

Each server machine spec was roughly as follows

Dual Quad-Core 3.2GHz
32GB RAM
2TB HDD (1x 256GB SSD Drive, 4x 450GB 15k SAS Drives)
QuadPort NICs (with teaming to increase capacative speed to the servers)

Each SAN was as follows

HP X3800 G2 (maxed out with HDDs)

What are you running on all of your servers to warrant so many for a school/academy?? Most I have seen in a school academy was 15 servers and that was because each role was virtualised out to an individual virtual server.

I have around 26/27 servers and can never remember the exact amount and topped out at 32 when I was transferring roles.

Imho it Depends on the amount of clients - We have around 900 clients so the file servers need the users distributed between them as We tend to have entire year groups logging in at one which meant that server ground to a halt while the rest worked perfectly

thats fair enough if each year group has their own server but that tops out at 7 servers....

Quote:

---

Originally Posted by nephilim

thats fair enough if each year group has their own server but that tops out at 7 servers....

---

We have 5 for data with year groups split over 3 servers. But then there are staff shares/student shares,dcs (2), av server/ webserver cluster (2), firewalls, proxys/terminal servers and the like. Can easily manage 40 servers if needs be.

I would be interested to here more info on the san as that sounds like a hefty peice of kit!

Fileservers, Sharepoint 2x WFE, 2 x App, 1 x SQL,
TMG, NG (network guardian), App servers etc etc. I'm sure you can figure out what each one does from the diagram below. It depends on the services you offer and the seperation you require. Seperation = simplicity = repeatability = stability attempting to achieve CIA. Restoring/losing 1 server only affects one service. If you have a virtual infastructure why not use it. It costs nothing to add a server (assuming you have Datacenter processor licenses) and an appropriate server specific license. Yes it adds a little complexity but we're all big boys and girls, I'm sure we can handle it.
Attachment 11771

Not questioning if you could handle it in the slightest, I'm sure you wouldn't have had them all if you couldn't!

Quite a remarkable thing, I would love to come and see it in action!

You are quite welcome to come. We are no different from an other comprehensive/academy, 1970s buildings a diminishing budget and a power supply that can barely cope. The only difference is that here in Rotherham several of us have walked the same path, shared our knowledge and have had the support to put things in place (at a school level that is). It started with one and grew from there. Other will, I guess, follow when the time and budgets allow.

First rule of IT club "talk to other schools about IT"

If you come, you will see what can be achieved with a relatively small investment over a protracted period of time. We are by no means the best but like everything you keep learning and improving, when you feel there is nothing left to learn it's time to move on. Personally, I am some way from that point.

The DS3500 is an "entry level SAN" whatever that means. It has claimed IOPS of 40,000 (turbo mode) supports 96 drives of SAS, SATA ranging from 15K SAS to 7,500 SATA. It takes 12 drives per enclosure with SAS cable connectivity between enclosures. It supports fibre channel (daughter card 4 ports per controller - so you can have a max of 4 hosts directly attached - I will have 3 plus one for a fibre SAN switch for Veeam), SAS (2 per controller - max of 2 hosts) It has a maximum of 2Gb per controller of cache (not particularly large) or iSCSI (4 port daughter card) and performs well in a school of 1650 staff and students.

I sound like an IBM salesman!

My existing SAN (which is due for replacemnt) is a DS3400 with 3 EXP3000 enclosures, dual controller populated with 48 300Gb 15K SAS drives, 2 x SAN16B fibre switches. It has 4 hot spares (we have had 4 drive failures in 5 years - hot spare kicked in each time, one went unnoticed for 2 weeks) and the rest is carved up into a variety of RAID 10/5 tier 1 and 2 LUNS. Usable capacity is about 7Tb, The other SAN (exactly the same as above but with 450Gb 15K SAS drives has 5Tb of usable space ie total of 12Tb hence the reluctance to buy the Tintri.

Anyone interested in the DS3400 above, it will be on sale in April (DS3400, 3 x EXP3000s, SAS cables and all 48 x 300Gb 15K SAS drives) if anyone wants to make me a reasonable offer.

Quote:

---

Originally Posted by Dave_O

Yes I think so, no more OS or Office moves for another 7 or 8 years. I'll be long gone by then!

---

No volume agreement? Much of the difficulty with training and migration occours because of jumping in huge leaps infrequently rather than a smoother little jump inbetween each step.

Ah ok - when you said it supported 96 drives i was thinking some beefy rack - didn't realise it was seperate enclosures. I've got a HP P2000 fiber channel which has a max iops of about 37000 and sounds very similar to what you have mentioned above. Although it didn't cost half that! I think the initial cost was around 13k for the San/fiberswitch and Disks. I then paid 4.5k for an additional enclosure including disks (12 450gb 15k sas). I will be adding an another enclosure next year to this and then in 3 years replacing this. I have around 8tb usable space in both which will go up next year to about 12. Overally i think it would have costs a lot less then the IBM Kit.

Also you mentioned that you would be getting the IBM server - from the spec you mention they do seem expensive. I've just purchased 2 x HP DL360 G7 with dual 6 core Xeons and 24gb ram for 6.5k for the both.

So San with enclosures - 20k
3 x hosts - say 10k
total 30k Divide that by 5 would give you a overall running cost PA for around 6k meaning you have 5k pa to spend on other things (25k over the 5 years)

plus you can't fault the HP proliant DL series kit. All the stuff i have worked with is great and even if i've had parts fail i've always had HP swap out the parts next day.

HP switches... fine HP servers... fine, HP entry level storage not had a good experience of them (MD series). each to his/her own. It's all based on your own experience. The value of the equipment, as I eluded to in the original post, isn't just about money. You don't make it clear if the costs are based on a 5 year warranty or that you just run the kit for 5 years If you factor these things in and the relative size/capacity my guess is they will not be much different. The servers are spec'ed with 2 x 6 core 2.99GHz processors, 48Gb memory, 6 x nics, 2 x 8Gb HBAs (the SAN is 8Gb fibre), 2 x power, no HD, bootable USB, 5 years warranty. As to the OS thing, we have EES but why keep changing so frequently, people want stability and predictability. Other than compatibility with the underlying OS and integration with other packages eg Sharepoint support from SIMS, there has to be a business need to change. eg SEN are still using Starspell 1 or 2! written I thnk as an 8 bit app and still good now. You have to ask who is driving the need for change is it us (IT) or them (staff, SLT). You only have to look at my other post on VDI to see the mistakes I have made in the name of progress.

It isn't all about price I agree. Those servers mentioned above included an extra 4 port network card so 8 nics overall. And 1 8gb hba (as I purchased one in the summer to put In an older box to test with) doubling the RAM would probs be an extra 2k overall.

I have to say I've never had a problem with the P2000 (touch wood). The Dl360's actually come with a use slot internal for the OS and also an SD card slot on the mobo

So with the cost of the extra 8Gb HBA roughly the same price.

The DS34/3500 and the HP SAN look very similar both visually and in terms of specification (although the DS3500 does have 2Gb of memory per controller). One of the things that would make the HP unsuitable for our new design is the fact that it only has 2 FC interfaces per controller meaning I would have to add the cost and complexity of 2 x SAN switches, which is something I am trying to move away from (more stuff to go wrong and therefore less to spend on the important things like the SAN itself). My guess is that it is no accident that the two are so similar and as I said previously I would guess that spec'ed identically they would not be far apart in cost. Still this has been quite useful in that when tenders come around in April, if IBM know that we will be getting quotes for something like the P2000 as a possible alternative this may influence their pricing somewhat.

It was 6k for 2 severs and 1 hba card so overall you would be looking about 10/11k for 3 hosts vs 21k.

We do have a fiber switch and only the one controller is active at once.

I run 18 servers from the san (2 exchange, 2 sql) and the utilization is about 7/8% of its capacity.

I will be moving the file servers to it at Christmas (2 to start with then a further 3 later on) and I'm going to monitor it closely with regards to performance and the bandwidth of the 2 8gb channels.