kmount

Hi, This may help you weigh up your options - Understanding the Impact of IOPS on Backup Performance: Best Practices RAID10 preferred for performance reasons, but be careful assuming you can lose multiple disks .... it depends which disks! ... if it happens to be the disks in the same group (ie both AA rather than an AB) you will have problems. There are tactics to mitigate against some of this but don't think it means you can lose half of your drives and still survive. (you could if it was a single A B C D etc but if it's an A and A it could spell disaster) - A fun explanation of this from this chap: A good explanation of this - hardware - Which RAID solutions can handle 2 disk failures - Server Fault The big challenge here is finding the balance between capacity and performance, both during backup, and also during recovery especially if you use instant recovery and want to offer reasonable performance during a disaster. SureBackup etc will also thank you for good performance. RAID0! The external company has set you up for a fail and I'm glad you've got wind of it and are putting it right.

Very good news indeed.

Evening, Normally I budget for an hour/two on an upgrade, the counter has a fixation with 80 minutes whenever I kick one off. (We have a few 3300's and one or two 3100s left) - I'd definitely get the upgrade done at your convenience just to keep on top of it. If all set up properly it should be fairly painless (in the future, it might have a hissyfit now) As to digging through Unisphere lets get a few bits to start with: - Storage Tab, VMware Storage (if empty, look in iSCSI Storage), and you should see your datastores listed, click on "Details" and in the top right hand corner it will say which "storage server" it is running on. (note that down for each datastore) - Settings Tab, iSCSI Server Settings, and you'll see a list of the iSCSI Servers configured along with the all important "Storage Processor" column - that combined with the above should be enough for us to deduce where each datastore is running - System Tab, System Performance, lets get a look at those graphs please, it should help us see how busy each SP is Cheers! Kim

Hi Nick, No need to feel guilty chap, always happy to help the community. I was a network manager for a number of years, now I look after technical services for a services company prodominantly around this kind of technology. Teejay's shout about the SIMS reindex patch is good, there's always going to be cause and effect here, the latency may be an effect rather than a cause. Good luck with the serial cable, getting that hooked up will hopefully get Unisphere sorted out again and then you can confirm if it's the shared SP (lets hope it is, narrows down our search) and the reindex patch is definitely worth running.

Hi there, Sorry for the delay in replying. vCenter (at the request of Veeam) taking yonks to delete snapshots is very very worrying, that is very indicative of an underlying issue. One thing I was curious about, does it only take a long time to remove the snap from the SIMS VM or from other VMs too? (busy VMs can make snapshot removal take longer because the snapshot can be potentially rather large). The VNXe doesn't support VAAI either so storage operations need to run up and down the wire but I'm not convinced that is relevant right now. Those read latencies are pretty high, I'd be quite concerned about them.. Now, lets get cracking to figure out what's happening... I can see DS1 & 3 are showing latency where DS2 & 4 aren't. I'm going to go out on a limb here but I suspect we're going to find that DS1&3 are on one of the controllers(SP) and DS2&4 is on the other one. If I'm right in that thinking I suspect we're going to find that it could be as simple as a CPU issue on the SP but once we get into Unisphere we'll know more. --> A reboot of the SP hosting DS1&3 if I'm right that they are together with 2&4 on the other might be a sensible move; if the multipathing is set up properly you shouldn't feel the reboot. (we don't) Can we drill down to get the iops for each of the VMs? (or even datastores) -- interestingly right now you're seeing high read latency but reasonably OK write which is quite unusual because it's normal to have some read cache in place to try and 'offset' a chunk of that. I'm confident we'll get to the bottom of this though! Cheers, Kim

You can indeed assign more than you have physical cores, that's part of the beauty of virtualization. - The caveat however is in understanding the below: If you assign a VM 8vCPU and it wants to do _anything_ it will have to wait until the host can find it 8 pCPU cores (or threads) upon which to execute the task. This is pretty bad if the VM just wants to say "hi!" Now, having one big VM and several little VMs is OK too because the scheduler on the host will be able to pop those 1-2 vCPU machines in and out to make space for your 8vCPU one but if you happen to have 16 CPU cores/threads to play with and you have 3 x 8vCPU VMs you'll find that only two of them can ever get service at a time with the third being effectively held still. It's not quite as black/white as that though as we're talking operations that occur in seconds. ESXi also likes a bit of spare core for itself to do the magic. - A long but useful outline is available - http://www.vmware.com/files/pdf/techpaper/VMware-vSphere-CPU-Sched-Perf.pdf So, you're 4:1 is fine, to be honest you'd likely get more but you 'up' your chances by being careful with vCPU allocation due to the scheduling, it's rare we find a VM that needs more than 4vCPU to be honest but of course it does happen. (we typically split roles out if we need more grunt than 4vCPU) I think your plan of action is solid, get the SAN sorted so we can see what it's saying about life. Something else I forgot to mention is to make sure you're not running any of your VMs in snapshots as that will increase your resource consumption (storage i/o side) as it may need to pick bits from layers.

Hi there, No worries, I can understand it's frustrating with hassle going on. First thing's first, look at the back of the SAN and you'll know pretty quickly if it's dual or single controller, it'll either have two 'sides' each with their own power/network etc or it'll have just one side and big space where another one could be! I suspect those 600GB's are 15k, (they certainly are in our VNXe's) so performance on that for a small/medium workload should be OK though RAID5 may play in but if it worked fine before and had been fine until recently lets not start SAN bashing just yet. Not being able to get into Unisphere is a worry, it could be something as trivial as a memory leak or such that could be ebbing away at your performance, or it could be something completely different! (first step here is to get onto EMC Support and get them to diagnose/resolve that problem). Have a look at https://pubs.vmware.com/vsphere-51/index.jsp#com.vmware.vsphere.monitoring.doc/GUID-44252CB8-5561-488A-A8CE-CF05C8F584BA.html & https://pubs.vmware.com/vsphere-51/index.jsp#com.vmware.vsphere.monitoring.doc/GUID-92C91273-F466-4B51-89CC-C7064E6171CE.html -- Appreciate it's a pretty old link but should be useful to at least see if the SIMS box (vs the others) is generating loads of I/O (i.e. whether it is causing/contributing to the issue) or whether it's just a victim. Going back to your VM spec, you say you've got 18 physical processor cores, can you confirm what other vCPU allocations you have on the same host? (i.e. if the SIMS box is 8vCPU how many others are allocated and importantly, in what quantities) Might also be worth looking at the performance snapshots to look at the CPU RDY and CPU WAIT on the host + VMs to see whether the host or VM is having problems getting a slice of the pie but we can come back to that. Getting into the Unisphere would be good to see what kind of workload it is reporting.

What disks are in the DPE/DAS' on it and what RAID level? Most importantly, if you look in Unisphere what kind volume performance are you seeing? If you're not sure where to look let me know and I'll pull up one of our VNXe's to check. On the vCenter console can you see what kind I/O your VMs (combined & individually) are generating? PS. Is it a single or dual service processor VNXe? Are your datastores spread over both SPs if it's the later. Are you noticing performance issues on other VMs or only this one because of the 100% CPU?

What spec/config is the SAN? How are the host's connected to it?

Out of interest ... If you mkdir /blah can you mount /dev/md-anything into /blah and have it complete without crying?

What's in mdadm --detail /dev/md122 and /dev/md124?

Ouch, that's proper angry. Though in context it may not be because for example md122 and md124 are the same size, as is md123 and m126 so it may just be a bit confused. Do you or does anyone have a frog box they can get a cat /proc/mdstat from to see what it *should* look like? I presume pulling a disk doesn't help when booting? (try pulling one and booting, if no joy reseat and pull the other and boot)

What does 'cat /proc/mdstat' show?

If you ask for a password reset it sends you your password in cleartext. (even if you update it and then try it again it *knows* the new one so not just a case of it keeping it post-migration until it is changed to a salted hash/etc). I'm sure its something now Impero know they'll get sorted. I had to laugh at what my password was when it came through

Hi there, You could look at Backup Assist, it's pretty cheap (circa £155 less 30% for Edu a server) Though if your license (including service?) is still valid with Symantec you could get them involved to help sort it out. Cheers, Kim

Make sure you ask whoever you talk to to run a Capacity Planning exercise on your current physical + virtual servers to provide you with the evidence behind their specification on CPU/Mem/Disk+IOPs so you know it's based on real information and fit for your current + predicted future needs. I'd be very surprised if you need more than 3 hosts, the number of CPU cores and RAM capacities now make less hosts more grunt feasible. This is especially important when it comes to licensing VMware and View which both have 'Essentials' bundles which are aimed at 3 hosts (6CPUs) for a low cost. That fourth host is where it becomes expensive so I'd recommend ramping up the spec of your 3 to cover you at n+1 with some spare over the top. I have a customer running 50+ VMs across 3 hosts no problem at all, they're a HE/FE college and it runs a bunch of stuff. The CPU RDY%/WAIT% is the important bit when it comes to whether you need another host if you have VMs waiting for CPU; in this case it doesn't happen because there's plenty of grunt. Re Hyper-V 2012 R2, a lot of customers are starting to take it seriously and it will continue to gain market share for all of the reasons the posters above pointed out so definitely worth investigating your options. Definitely definitely go with a Capacity Planning exercise though, guessing isn't good enough! Have worked with Ergo in the past, nice chaps.

I had something similar the other day and it was to do with SQL Server at boot up making the firewall service time out. The event viewer should hold some clues on when it stopped and you can see what else was lurking around the same time. (and its not a pretty way forward but you could modify the service to automatically restart on failure)

I question your logic, I'd sell the missus and keep the router I tend to justify these kinds of purchases to the wife by using terminology she understands ... Netflix will be faster to buffer, Steam downloads will be faster etc. Looks like a nice router, but my current Asus equally ugly unit is doing fine.

*yoink* - thank you kindly.

What exactly doesn't work? What error are you seeing? There is/was a weird NTLM error bug that impacted the mac client but without the error it's hard to be sure.

Source please?

Nah, on VMware it behaves differently but it's all to do with nic's and deadlocks leading to resource queueing. Updating the drivers should be a good move for you I'd say.

https://downloadcenter.intel.com/SearchResult.aspx?ProdId=3356&lang=eng then pick Windows Server 2012R2 The VMware version numbers are different but there's some new ones there. Worth a punt as my problem was iSCSI too!

ooooh, the I350! I found a problem with this on some VMware hosts the other day! Check what nic driver version you're on, the latest on the Intel site has stopped my hosts being silly for 25 days and counting!

OT: Just clocked Sophos there, have you got it excluding where your Hyper-V VMs live yeah? The network drivers is a good shout, what nic's are they? How is the storage provided to your Hyper-V environment?