Dave_O Posted October 22, 2011 Posted October 22, 2011 You've probably seen some of my previous posts on the virtualisation thread. Here's an update on where we are and what we are going in the virtual world. I hope it's of some use to you. Background to recent changes at the chalk face The move to Windows 7, Office 2010 and IE9 has been an interesting one, throwing up all manner of issues we would not have even considered possible just a few weeks ago. All seem to be resolved apart from an Impero issue with internet blocking (or not) and multiple tab opening and SIMS not working well with Office 2010 (there's a suprise). System Centre works a lot better with Windows 7 machines. We thought MAK activation might be an issue (we decided to use this in favour of KMS can't remember why) but ended up being easy. Sharepoint 2010, Exchange 2010 Lync 2010 all working nicely. In fact everything seems to be OK. Was this a good move? Yes I think so, no more OS or Office moves for another 7 or 8 years. I'll be long gone by then! What's all this got to do with virtualisation? Quite a lot as it's the front that drives the rear (if you'll excuse the double entendre). That and the growing and ever changing needs of the staff and students. The school's reliance on school hosted services, therefore, (our decisions as other solutions we unreliable and inflexible) makes the SAN and ESX host decisions critial. Virtualisation changes Times are hard and money short even for academies. Two of our ESX's and the primary SAN are up for renewal in April next year after 5 years of perfect service. This time I'm going for simplicity. As the IBM kit we have is so reliable, I am moving to just 3 ESXs each with 48Gb memory, consolidating to a single SAN (rather than 2/3) and removing the SAN switch. I will, however, be using one of out older SAN switches to add our backup solution, Veeam, but the ESXs will access the SAN directly. Why are we reducing to 3 ESXs? We already run 40+ virtual servers and I cannot envisage us expanding much more over the next 5 years other that file, mail and intranet services which rely on storage capacity rather than ESX memory. I defy anyone to show me a school in the UK that needs more than this! The choice of server is a no brainer... IBM x3650 M3. Good reliable service - worth every penny. We will be buying with 5 years warranty which, to be honest, we have only ever used once on a faulty CD drive 6 months after purchase. You can probably get faster but I'll go for the reliability every time. We expect to pay about £7,500 per unit (£1,400 pa) The choice of SAN, though, is a little more difficult. I have been interested in the new technologies around at the moment, in particular Tintri (a demonstration of which we are going to next month) offering a 4U box with 12Tb made up of SSDs and SATA. The claimed IOPs are impressive, I only hope the price and reliability are equally impressive. Our storage requirements, unfortunately, are large and a single Tintri unit would only just be big enough for what we have now so we would have to consider buying 2 units. This, I think, will put the solution beyond our means. The replacement for our IBM DS3400 will probably be the IBM DS3512 wich has more capacity (supports 96 drives) and more cache. We will probably buy the unit with 4 expansion trays (populating only 3 of these fully with 600Gb 15K SAS drives in the first place) and turbo performance. I did look at remote mirroring but to implement that would cost in the region of £13,000 (including the hardware) not cost effective given our SLA. I expect to pay no more than £35,000 for the SAN. Which works out at around £7,000 per year, interestingly that is 4 x less than we pay per annum for our internet connection (yet another story soon to be resolved)! It's a hell of a lot more reliable and delivers a damn site more services! The total cost of maintaining the SAN + 3 ESXs is therefore £11,400 pa. Not a huge amount in the scheme of things. Whatever direction we take in the storage arena our rack space rather than decreasing, is actually increasing as we consolidate more of the physical kit.
featured_spectre Posted October 22, 2011 Posted October 22, 2011 40+ servers??? What on earth? Does each server run a separate role? On Server 1 I had the following...DHCP, DNS, ADUC, Print Server, Kerboros - For Admin/Staff On Server 2 I had the following...SQL, Exchange 2010 On Server 3 I had the following...DHCP, DNS, ADUC, Print Server, Kerboros - For Students On Server 4 I had the following...IIS 7.5, KMS Server (for all devices), ADCS, ADLDS, Radius - For all people in the school On Server 5 I had the following...File Services - For all people in the school SAN 1 - File Storage SAN 2 - Backup Each server machine spec was roughly as follows Dual Quad-Core 3.2GHz 32GB RAM 2TB HDD (1x 256GB SSD Drive, 4x 450GB 15k SAS Drives) QuadPort NICs (with teaming to increase capacative speed to the servers) Each SAN was as follows HP X3800 G2 (maxed out with HDDs) What are you running on all of your servers to warrant so many for a school/academy?? Most I have seen in a school academy was 15 servers and that was because each role was virtualised out to an individual virtual server.
glennda Posted October 22, 2011 Posted October 22, 2011 I have around 26/27 servers and can never remember the exact amount and topped out at 32 when I was transferring roles. Imho it Depends on the amount of clients - We have around 900 clients so the file servers need the users distributed between them as We tend to have entire year groups logging in at one which meant that server ground to a halt while the rest worked perfectly
featured_spectre Posted October 22, 2011 Posted October 22, 2011 thats fair enough if each year group has their own server but that tops out at 7 servers....
glennda Posted October 22, 2011 Posted October 22, 2011 thats fair enough if each year group has their own server but that tops out at 7 servers.... We have 5 for data with year groups split over 3 servers. But then there are staff shares/student shares,dcs (2), av server/ webserver cluster (2), firewalls, proxys/terminal servers and the like. Can easily manage 40 servers if needs be. I would be interested to here more info on the san as that sounds like a hefty peice of kit!
Dave_O Posted October 22, 2011 Author Posted October 22, 2011 Fileservers, Sharepoint 2x WFE, 2 x App, 1 x SQL, TMG, NG (network guardian), App servers etc etc. I'm sure you can figure out what each one does from the diagram below. It depends on the services you offer and the seperation you require. Seperation = simplicity = repeatability = stability attempting to achieve CIA. Restoring/losing 1 server only affects one service. If you have a virtual infastructure why not use it. It costs nothing to add a server (assuming you have Datacenter processor licenses) and an appropriate server specific license. Yes it adds a little complexity but we're all big boys and girls, I'm sure we can handle it.
featured_spectre Posted October 22, 2011 Posted October 22, 2011 Not questioning if you could handle it in the slightest, I'm sure you wouldn't have had them all if you couldn't! Quite a remarkable thing, I would love to come and see it in action!
Dave_O Posted October 22, 2011 Author Posted October 22, 2011 You are quite welcome to come. We are no different from an other comprehensive/academy, 1970s buildings a diminishing budget and a power supply that can barely cope. The only difference is that here in Rotherham several of us have walked the same path, shared our knowledge and have had the support to put things in place (at a school level that is). It started with one and grew from there. Other will, I guess, follow when the time and budgets allow. First rule of IT club "talk to other schools about IT" If you come, you will see what can be achieved with a relatively small investment over a protracted period of time. We are by no means the best but like everything you keep learning and improving, when you feel there is nothing left to learn it's time to move on. Personally, I am some way from that point. 1
Dave_O Posted October 23, 2011 Author Posted October 23, 2011 The DS3500 is an "entry level SAN" whatever that means. It has claimed IOPS of 40,000 (turbo mode) supports 96 drives of SAS, SATA ranging from 15K SAS to 7,500 SATA. It takes 12 drives per enclosure with SAS cable connectivity between enclosures. It supports fibre channel (daughter card 4 ports per controller - so you can have a max of 4 hosts directly attached - I will have 3 plus one for a fibre SAN switch for Veeam), SAS (2 per controller - max of 2 hosts) It has a maximum of 2Gb per controller of cache (not particularly large) or iSCSI (4 port daughter card) and performs well in a school of 1650 staff and students. I sound like an IBM salesman! My existing SAN (which is due for replacemnt) is a DS3400 with 3 EXP3000 enclosures, dual controller populated with 48 300Gb 15K SAS drives, 2 x SAN16B fibre switches. It has 4 hot spares (we have had 4 drive failures in 5 years - hot spare kicked in each time, one went unnoticed for 2 weeks) and the rest is carved up into a variety of RAID 10/5 tier 1 and 2 LUNS. Usable capacity is about 7Tb, The other SAN (exactly the same as above but with 450Gb 15K SAS drives has 5Tb of usable space ie total of 12Tb hence the reluctance to buy the Tintri. Anyone interested in the DS3400 above, it will be on sale in April (DS3400, 3 x EXP3000s, SAS cables and all 48 x 300Gb 15K SAS drives) if anyone wants to make me a reasonable offer.
SYNACK Posted October 23, 2011 Posted October 23, 2011 Yes I think so, no more OS or Office moves for another 7 or 8 years. I'll be long gone by then! No volume agreement? Much of the difficulty with training and migration occours because of jumping in huge leaps infrequently rather than a smoother little jump inbetween each step.
glennda Posted October 23, 2011 Posted October 23, 2011 Ah ok - when you said it supported 96 drives i was thinking some beefy rack - didn't realise it was seperate enclosures. I've got a HP P2000 fiber channel which has a max iops of about 37000 and sounds very similar to what you have mentioned above. Although it didn't cost half that! I think the initial cost was around 13k for the San/fiberswitch and Disks. I then paid 4.5k for an additional enclosure including disks (12 450gb 15k sas). I will be adding an another enclosure next year to this and then in 3 years replacing this. I have around 8tb usable space in both which will go up next year to about 12. Overally i think it would have costs a lot less then the IBM Kit. Also you mentioned that you would be getting the IBM server - from the spec you mention they do seem expensive. I've just purchased 2 x HP DL360 G7 with dual 6 core Xeons and 24gb ram for 6.5k for the both. So San with enclosures - 20k 3 x hosts - say 10k total 30k Divide that by 5 would give you a overall running cost PA for around 6k meaning you have 5k pa to spend on other things (25k over the 5 years) plus you can't fault the HP proliant DL series kit. All the stuff i have worked with is great and even if i've had parts fail i've always had HP swap out the parts next day.
Dave_O Posted October 23, 2011 Author Posted October 23, 2011 HP switches... fine HP servers... fine, HP entry level storage not had a good experience of them (MD series). each to his/her own. It's all based on your own experience. The value of the equipment, as I eluded to in the original post, isn't just about money. You don't make it clear if the costs are based on a 5 year warranty or that you just run the kit for 5 years If you factor these things in and the relative size/capacity my guess is they will not be much different. The servers are spec'ed with 2 x 6 core 2.99GHz processors, 48Gb memory, 6 x nics, 2 x 8Gb HBAs (the SAN is 8Gb fibre), 2 x power, no HD, bootable USB, 5 years warranty. As to the OS thing, we have EES but why keep changing so frequently, people want stability and predictability. Other than compatibility with the underlying OS and integration with other packages eg Sharepoint support from SIMS, there has to be a business need to change. eg SEN are still using Starspell 1 or 2! written I thnk as an 8 bit app and still good now. You have to ask who is driving the need for change is it us (IT) or them (staff, SLT). You only have to look at my other post on VDI to see the mistakes I have made in the name of progress.
glennda Posted October 23, 2011 Posted October 23, 2011 It isn't all about price I agree. Those servers mentioned above included an extra 4 port network card so 8 nics overall. And 1 8gb hba (as I purchased one in the summer to put In an older box to test with) doubling the RAM would probs be an extra 2k overall. I have to say I've never had a problem with the P2000 (touch wood). The Dl360's actually come with a use slot internal for the OS and also an SD card slot on the mobo
Dave_O Posted October 23, 2011 Author Posted October 23, 2011 So with the cost of the extra 8Gb HBA roughly the same price. The DS34/3500 and the HP SAN look very similar both visually and in terms of specification (although the DS3500 does have 2Gb of memory per controller). One of the things that would make the HP unsuitable for our new design is the fact that it only has 2 FC interfaces per controller meaning I would have to add the cost and complexity of 2 x SAN switches, which is something I am trying to move away from (more stuff to go wrong and therefore less to spend on the important things like the SAN itself). My guess is that it is no accident that the two are so similar and as I said previously I would guess that spec'ed identically they would not be far apart in cost. Still this has been quite useful in that when tenders come around in April, if IBM know that we will be getting quotes for something like the P2000 as a possible alternative this may influence their pricing somewhat.
glennda Posted October 23, 2011 Posted October 23, 2011 It was 6k for 2 severs and 1 hba card so overall you would be looking about 10/11k for 3 hosts vs 21k. We do have a fiber switch and only the one controller is active at once. I run 18 servers from the san (2 exchange, 2 sql) and the utilization is about 7/8% of its capacity. I will be moving the file servers to it at Christmas (2 to start with then a further 3 later on) and I'm going to monitor it closely with regards to performance and the bandwidth of the 2 8gb channels.
Dave_O Posted October 23, 2011 Author Posted October 23, 2011 Yes I see. However I don't really understand how you can get a 4 nic card and 2 x 1 port HBAs in a 1 U DL360 G7 or am I missing something? Also are the dual controllers not setup as active/active otherwise what's the point in having 2 controllers? Our present setup has run all of our infrastructure for 4 and a half years, we monitor performance and latency on a regular basis and have found few, if any issues. SIMS runs slow on occasions but I've tried it both on physical and virtual and performance seems to be agnostic of the technology you run it on - my conclusion... SIMS is crap, but then you all know that anyway. You really can't put a price on reliabiity... well yes I can and have.
glennda Posted October 23, 2011 Posted October 23, 2011 I've got a 2 port hba card in them into the fiber switch ( HP 8/20q). The 4 port nic card comes with a half height adaptor so fits in aswell. Sims is rubbish ill agree with that! I've only been here 2 years and getting everything running realiably, reducing login times of 4/5mins to less then 1 was more important then drastic back end changes (enless needed) Now everything is running reliably We can look to do.more virtuliseation stuff which is where I am currently. My other virtual hosts are dl380's which will take 3/4 full height cards and a half height card. They come in very similar price/spec but the 380 is designed more with in server storage in mind with 8sff sas drive bays. Having two controllers is for failover. If one controller fails the hosts auto connect to the other ( We use Linux kvm and an Oracle file system cluster and multipath) . So you have no downtime and get 16gb to the san from the fiber switch
Dave_O Posted October 23, 2011 Author Posted October 23, 2011 You do well to take your time, reliability is the key word here. With reliability you can argue for replacement and expansion. SLT and governors knew the replacement was coming 2 years ago (mainly because I kept bringing it up at line management every 3 months - never underestimate the power of repeating yourself, it's when they don't know that problems arise). Experiences of IT at different schools vary widely (I am working with one school where login times are around 15 mins!), for us major problems are rare if they ever did happen staff would not really be well equiped to deal with them (possibly a bad thing). Our choice of desktop PCs has followed a similar path this time all Dell, few laptops (nipped that one in the bud early on), not the cheapest but they really do last for years. Our replacement cycle is now 6 years our oldest Dells (GX520s) work fine with Windows 7, we will continue to replace about 130 per year. Printer consumables £8,000 pa. Total spend pa is now down to about £120,000. If the school wants reliability & performance that's the cost oh yes and the IT support team's salary. Sorry I digress.
glennda Posted October 23, 2011 Posted October 23, 2011 That's very similar to what We get Although We will be hitting 220 a year replacement on a 5 year cycle with next summers less (because on the original plan of the previous next summer is the servers rather then desktops. The only failures We seem to have at the moment are air-con and internet which is out of my control! The air was serviced 5 weeKs before it failed ( I do have 2 but anything longer then 1/2 hours it get above a comfortable temp. And the internet was a hardware failure with hr's media converter!
Dave_O Posted October 23, 2011 Author Posted October 23, 2011 Same! Last weekend had the VDI server room at melt down due to aircon failure (both units!!). One of the IBMs recorded 45 degrees before shutting itself down (have changed the criticality levels down to 35 degrees). Internet! don't talk to me about internet, the LA supply ours. Last September...chaos, but not for much longer. I can't go into detail for legal reasons but things will improve both in terms of reliability and cost effectiveness. Internet is the last thing out of our control. As I've said in previous posts (perhaps controvertially) solutions bigger than a school don't work. This isn't isolationism, it's pragmatism and being on the sharp end of implementations that just don't deliver is not a good place to be. The whole change management process becomes rediculously protracted. A change I can make in just a few minutes takes days. So when I got "I'm sorry you can't use office web apps as we are on Sharepoint 2007 and will not be changing any time soon", I lost the will to live and did it myself. Office 365 and live at edu (or whatever its called) equally disappointing
glennda Posted October 23, 2011 Posted October 23, 2011 I made sure ours didn't get to that - shutdown non critical systems to reduce heat etc. Ours peaked at 27. But is now happily running sub 20. Although I've now been approved to get roomalert with a few sensors to alert me to the issue in the future.
irsprint84 Posted October 23, 2011 Posted October 23, 2011 Same! Last weekend had the VDI server room at melt down due to aircon failure (both units!!). One of the IBMs recorded 45 degrees before shutting itself down (have changed the criticality levels down to 35 degrees). Internet! don't talk to me about internet, the LA supply ours. Last September...chaos, but not for much longer. I can't go into detail for legal reasons but things will improve both in terms of reliability and cost effectiveness. Internet is the last thing out of our control. As I've said in previous posts (perhaps controvertially) solutions bigger than a school don't work. This isn't isolationism, it's pragmatism and being on the sharp end of implementations that just don't deliver is not a good place to be. The whole change management process becomes rediculously protracted. A change I can make in just a few minutes takes days. So when I got "I'm sorry you can't use office web apps as we are on Sharepoint 2007 and will not be changing any time soon", I lost the will to live and did it myself. Office 365 and live at edu (or whatever its called) equally disappointing We have internet away from Council now, forgot about all the problems used to have!
Dave_O Posted October 23, 2011 Author Posted October 23, 2011 Interesting... roomalert.. which version, which sensors, how much?
glennda Posted October 23, 2011 Posted October 23, 2011 The 4er with an extra digital temp/humidity sensor.
Recommended Posts
Create an account or sign in to comment
You need to be a member in order to leave a comment
Create an account
Sign up for a new account in our community. It's easy!
Register a new accountSign in
Already have an account? Sign in here.
Sign In Now