Spindles / SAN

Hi,

what's peoples thoughts on SAN / LUN sizing and VM placement nowadays? If you were to setup a school from scratch, using what you have learnt from the last few years, what would you do differently:

How you would you setup your SAN?
How would you size your LUNs?
How would you decide where to place your VM's?
Would you dedicate for SQL or Exchange data and log files?
etc.

Basically i'm reviewing our SAN setup, and i've read a lot recently but i'm very interested in your opinions especially given you run school-style workloads.

Quote:

---

Originally Posted by oxide54

Hi,If you were to setup a school from scratch, using what you have learnt from the last few years, what would you do differently

---

There seems to be a tendancy for schools to emphasise processing capacity over storage capacity, but most schools need lots and lots of reliable/cheap/fast storage capacity. My ideal infrastructure setup for a school network consisting of a bunch of Windows workstations joined to a domain would be:

One Dell PowerEdge R710 or similar, with as many processors and as much RAM as is practical / affordable and populated with reasonably fast harddrives in each harddrive slot. That would be licensed for Windows Server Enterprise, letting you run four Windows virtual servers. Use one as a Domain Controller, one as a print server, one for general applications and one for your MIS. It should also probably have a 4-port ethernet card to allow for a decent-sized network connection straight to the main switch, with the built-in ethernet port being assigned to a VM that handles your firewall. I'd then have at least a 10TB storage server, with a fast hardware RAID card, as the live file server.

Ideally, you would also have a second server as above in a separate location along with a 15-20TB backup server. The second server would mirror some of the VM images from the main server - the MIS, print server and applications server. You'd have a separate domain controller VM on the second server running on un-mirrored local storage - DCs can do their own mirroring. Also, the print server should have its own un-mirrored local storage for the print spool folder on each server. Otherwise, every byte written to the main server should be mirrored to the second server.

In normal operation, you'd probably have the applications server run on the second physical server to make use of the otherwise spare processing capacity. Ideally, your second physical server would have its own Internet connection - you could either mirror your firewall or, more effciently, have two gateway servers and split network traffic between the two.

Any remote desktop servers should be on their own dedicated bits of hardware - they're user workstations, and should be treated as such (deployed and installed by workstation imaging software).

The above setup should make for a system that gives as much uptime as is reasonable to expect from a school. If one Internet connection goes down the other can take over (if it's on a separate physical connection, exchange, etc). If one server location goes down for some reason (power outage, flood, fire, etc) the other can take over (assuming separate power, lack of simultanious fire/flooding, etc). A half-decent UPS and air-conditioning in each location is probably practical, but really you only need to provide enough uptime for the servers to shut down cleanly.

The 10TB storage server would just be for general user files, of course - if you want to store photos or video you should get another file server, with a matching backup server.

Quote:

---

Would you dedicate for SQL or Exchange data and log files?

---

If you have to run Exchange or a separate SQL server then you should have its storage mirrored. If the local storage on the main/mirror servers is split over enough disks you could assign a dedicated couple of disks via your RAID card to be just for the SQL / Exchange server.

okay, i already have a 5 node vmware cluster hooked up to SAN, i was specifically more meaning how to carve the SAN for best (better than now) performance,

If you want high performance you want to look at the ZFS filesystem with a [couple of] SSD[s]. That setup means as long as you have enough spindles for your read load there is no reason why you can't use inexpensive 7.2k rpm drives.

ZFS also comes with some tasty feature such as dedupe and compression which actually outperforms non-compressed. Also the way ZFS sets out its FS in a logical and layered way is brilliant.

I could well imagine compressed would outperform uncompressed if done properly, ghosting was always quicker using fast compression if the disk layer/cpu was quicker than the network

Quote:

---

Originally Posted by oxide54

i was specifically more meaning how to carve the SAN for best (better than now) performance,

---

How many spindles do you have in your storage server? If you have a single, central SQL server instance that hosts databases for everything else then it's probably worth assigning it a dedicated RAID-1 array to itself if you have the capacity. You might want to check your current SQL server to see if it's being held up by disk performance or RAM - I keep hearing that the more RAM you can give SQL or Exchange, the better it performs.

I'm not too sure about Exchange performance (if this was a school "starting from scratch" I'd avoid it), but our Exchange server seems to have decent enough performance to manage 30-50 simultanious users running as a single-processor VM with 4GB of RAM on a shared RAID array hosted by a QNAP device on an iSCISI connection. Again, assigning it more RAM would probably help. I'd probably aim for extra storage volume for an Exchange server - you want people to have enough room to keep most of their emails, and I figure a school should be able to have records available for at least the time it takes for a pupil to progress through the school (7 years?).

This is a good article.
Storage Top 10 Best Practices

In a nutshell, arrange your arrays vertically to spread the risk of failure over different boxes, Keep data files and log files on their own separate arrays, RAID5 is normally OK for data, RAID 10 or 1 is best for logs.

I used IOMeter to gauge the performance of different configurations. I think I've still got the performance logs and charts.

Quote:

---

Originally Posted by jinnantonnixx

RAID5 is normally OK for data

---

With today's larger harddrives, the chances are around 1-in-5 (I think) that if one disk conks out and you need to replace it and rebuild the array you'll find a read error somewhere on one of your other disks. I'd go for RAID-6, just to be on the safe side.

Quote:

---

RAID 10 or 1 is best for logs.

---

The advantage of a decent RAID card is that it provides a large RAM cache for disk read / writes. If you're writing log files you're always writing new data, so having a cache is a bit pointless - you just need decent disk performance. I wonder if the best option for a log file volume is a couple of high-performance drives in a softare RAID-1 array, avoiding the RAID card's cache? Or do RAID cards have a feature that allows you to turn off cache usage on a particular array?

Quote:

---

Originally Posted by oxide54

okay, i already have a 5 node vmware cluster hooked up to SAN, i was specifically more meaning how to carve the SAN for best (better than now) performance,

---

The simplistic answer is that better performance is obtained by more spindles and fewer VM's per iSCSI volumes. But since you don't give any information about loads, architecture or environment a simplistic answer is all that is really possible.

The real answer is it depends on what the current demands are, what your hardware is capable of servicing and how your set-up handles a situation where it runs out of steam for arbitrary periods of time and has to start queuing requests, if indeed that ever happens. You can't after all get better performance from a system that is never being asked to deliver even 50% of it's capability. iSCSI involves an overhead which leads to advice that you should limit individual vmfs file systems to 5-10 guests. That iSCSI overhead can be compounded as you attach more hosts to the storage so if you have clusters or are just sharing volumes between hosts a lower number of VM's per volume is better. Worth arranging were possible for guests sharing the same vmfs volume to be live on the same host to minimise contention but the importance of that ultimately depends on the loads the Guests are making.

You don't say which Virtualisation software you are using, so some of that will apply but vmfs constraints may be different to the filesytems Xen uses. At the SAN level more spindles will give you a better ability to handles throughput in terms of IOPS and Mbs and that is further modified by the RAID chosen. RAID 6 will have a high write overhead so will have lower throughput for a given number of spindles than Raid 0, but RAID 0 would be fragile and increasing spindles would have a worrying effect on probability of array fail and possible data loss. What I'd choose would depend on the application and how tollerant that might be to short term data loss. Financial transactional systems I'd rather sacrifice performance than have to start unpicking the lunchtime dinner money transactions but some lump of storage hosting videos - I'd probably argue they could lose a days data and not be in too much of a huff.

It would also depend on your storage requirements, So unless you have literally a ton of money and go SSD completely on your SAN, what speed disks are you you using ,5900, 7200, 10000, 15000 they will all make a difference, a decent SAN solution should allow you to group different Disk speeds into a seperate groups and then allocate these disk groups. SO go for the high speed disk on your SQL stuff, and low speed stuff for things like photos etc.

So it really is a case of where do you want the most speed unless you are gonna SSD all the way.

Interesting question. We have just replaced our old SAN with a new IBM V7000 made up of 2 trays dual controller, 8Gb cache per controller, 8Gb fibre 40 x 2.5" 600Gb 10K SAS drives, 2 x 2.5" 200Gb SSD drives. Theoretical IOPs is supposed to be abour 56,000 but don't think this is the case in the real world. Under normal use with roughly 60% of our stuff on, it registers about 1,400 IOPs in on the graphical monitor, when doing SAN based backups (Veeam) it goes up to anything around 15-20,000 IOPs. These figures are based on only a few days observation and may change over time. It is carved up as follows:-

Pool 1 - 1 volume, 8 drives RAID 1+0 2.2Tb usable space. IBM easy tier enabled - this uses the 2 SSDs to cache "hot" blocks, does this automatically over time. Don't know what the performance improvements will be, will have to wait and see. (On this volume is Exchange, Sharepoint and SIMS - read and write stuff). To be honest the whole SQL and Exchange log issue is not really relevant at a school level as long as the performance of the volume is good is will make very little difference
Pool 2 - 1 volume, 12 drives RAID 6 5.6Tb usable space. (Fileservers, misc inc web servers etc - mainly read some write)
Pool 3 - 1 volume, 12 drives RAID 6 5.6Tb usable space. (Fileservers, SCCM, misc inc web servers etc)
Pool 4 - 1 volume, 6 drives RAID 5 2.6Tb usable space. (Misc inc web servers etc - mainly read)
2 x Hot spares.

In short large volumes using high numbers of spindles different raids on pools for different work load types. Doesn't need to be any more granular or sophisticated than that. You can put smaller volumes in the pools and still benefit from the high spindle count though.

vSphere 5 can see these large datastores but you can still only use a 2Tb (max) drive in a Windows server obviously you can chain these together to make larger drives if you want.

Was it worth the money to replace? I really do have to question whether we in schools need to be thinking about storage that can pull anything more than 20,000 IOPs. From my observations we don't use it. When I first got my DS3400 back in 2006 with IOPs of up to 10,000 I was a bit worried that it might not be capable of running all the stuff I put on it. Time has shown that my worries were unfounded. Given our mix of servers (and we have a lot running everything from Exchange 2010 through Sharepoint 2010, SIMS (ug!) to MLS to room booker) it performed well. However it was getting to 5 years old and it was time I had a new toy but at around £38,000 its not cheap. Along with the SAN, ESX replacements will mean that desktop replacement will suffer this year. More than anything it is this that will probably tarnish our image as providers of a reliable IT service as the desktop kit gets older. As we all know its the end user experience that affects your image not the quality and reliability of the service you supply.

For those who have followed my VMWare View threads, yes I have tried View on the new storage (with SSDs) and the performance, although better is still nothing to write home about. Will stick to terminal services through thin clients for the moment.

Dave O

excellent answer, I personally do think the SQL is an issue but only with SIMS some of the queries do take a long time to execute running certain types of reports.

I will be replacing the san in the next year, partly performance related and partly capacity but also for the benefits the warranty brings as replacement disks get expensive when a product is end of life, and once those disks have been spinning 24/7 for a few years you begin to question how much longer they will last, we haven't seen any failures on the SAN itself but have on both NAS devices of the same age from the same vendor ...

but I won't be replacing the SAN now to cure a performance bottleneck if it can resolved by reconfiguration.

I'm currently shrinking all the VM's to the smallest size possible (with some breathing space) so as to enable some LUN re-jiggery.

SAN disk failures

Over the approximately 100 disks in 3 SANs over 5 years we have lost about 6 disks(hot spare taken over with no loss of service). Other schools in the authority who have the same kit have not lost as many if any. I think our school is in a particularly dusty area. Other evidence for this is the amount of time we spend cleaning projectors!

SIMS queries

The SIMS SQL query issue is (in the main) due to the poor query optimization, which to be fair is not unsurprising as SIMS is primarily a transactional database. Most of the queries from the reports are ad-hoc multi-table affairs. It would be better if they did the reports from the discover database. OK the data is not as up to date as the live SIMS data but for most reports this is not an issue. Pushing the data to a denormalised query optimised data warehouse like this (especially if it is on a different box) would speed things up measurably.

SAN replacement/reconfiguration

You are correct on warranties you cannot skimp on the SAN, too critical. A colleague of mine at another school in the authority with a couple of DS3400s is reconfiguring his SAN (he has enough spare space to move things around) to make fewer larger arrays/pools made up of more disks and larger volumes. If you have space to move stuff off the LUNs/volumes and reconfigure then I would do this.

Just an addendum to my previous post. Whatever you do, do not create a drive (eg for a file server) of exactly 2Tb. Yes I should have read the VMWare stuff more carefully because when you attach a disk of 2Tb you cannot snapshot it and all Veeam backups fail. I have had to do all manner of giggery pokery to correct this (no you can't shrink it back!) So the moral of the story is:-

When you add a disk to your VM make it less than 2Tb - I now make mine less than 2000Gb
You can add as many of these as you like and stitch them together to make more space just as long as each one is less than 2Tb

[I do this so you don't have to. Or at least you can have a go at doing it without screwing everything up]

As I keep saying, "You're never too old to learn." and "Lessons learnt the hard way are lessons never forgotten" - bit of a dig at a previous post in another thread. Which brings me to my last misquote "You can teach an old dog new tricks" at least this one you can.

Hope this helps

Dave O

I installed SQL IO on my SIMS server (virtual) and ran 4 tests - sequential read & write and random read & write over a 30 second period. The results are below. I would be interested in the results of others running the same commands (also given below):-

sqlio -kR -s30 -frandom -o8 -b8 -LS -Fparam.txt
CUMULATIVE DATA:
throughput metrics:
IOs/sec: 34867.99
MBs/sec: 272.40
latency metrics:
Min_Latency(ms): 0
Avg_Latency(ms): 0
Max_Latency(ms): 98
histogram:
ms: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24+
%: 99 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

sqlio -kW -s30 -frandom -o8 -b8 -LS -Fparam.txt
CUMULATIVE DATA:
throughput metrics:
IOs/sec: 34123.80
MBs/sec: 266.59
latency metrics:
Min_Latency(ms): 0
Avg_Latency(ms): 0
Max_Latency(ms): 84
histogram:
ms: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24+
%: 99 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

sqlio -kR -s30 -fsequential -o8 -b8 -LS -Fparam.txt
CUMULATIVE DATA:
throughput metrics:
IOs/sec: 36841.62
MBs/sec: 287.82
latency metrics:
Min_Latency(ms): 0
Avg_Latency(ms): 0
Max_Latency(ms): 23
histogram:
ms: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24+
%: 99 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

sqlio -kW -s30 -fsequential -o8 -b8 -LS -Fparam.txt
CUMULATIVE DATA:
throughput metrics:
IOs/sec: 10728.56
MBs/sec: 83.81
latency metrics:
Min_Latency(ms): 0
Avg_Latency(ms): 0
Max_Latency(ms): 52
histogram:
ms: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24+
%: 85 0 0 0 2 3 2 4 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0