Exchange running really slow

[Darylrese]

Hi

 

For the last 2 days or so our exchange has been grinding to a halt and outlook 2010 freezes and crashes, constantly displaying this message to users:

 

 

Pinging the server also seems worrying:

 

 

Exchange is a virtual server with 12GB allocated memory. Its sitting at a constant 11GB useage and doesnt fluctuate atall. Is it bottlenecking? What else could be wrong?

 

Our old exchange used to only have 4GB memory and coped fine! We upgraded to new servers in the summer, coming away from an RM system. Its been working fine for the last 3 weeks but has just started happening this week.

 

Tried cache mode, still the same if not worse.

Edited October 3, 2012 by Darylrese

[Domino]

Exchange tends to grab as much RAM as it can, so don't worry about the ysage too much.

 

Is this a 2010 deployment? one server?

 

Has it got multiple network cards, and do you have them trunked/linked at all?

[Darylrese]

Yep Exchange 2010 and only one exchange server. Yes i believe there are two NIC's trunked together.

 

Our store.exe is taking up 11GB out of 12GB which i believe may be slowing the server down. Going to take it offline later, bump the server up to 16GB and limit the store.exe cache size to something like 12GB, leaving 4GB for other applications. Hopefully that will improve things?

[nickbro]

Exchange 2007+ is very RAM intensive, unlike 2003-. It may be that it doesn't like running as a virtual machine, but I can think why. The ping timings look very bad, does it had a dedicated NIC on the virtual server or is it sharing it with other servers?

 

Have you applied the latest Update Roll Ups? I think we're SP2 Roll Up 3 now.

 

My Exchange 2010 server is constant with it's ping:

Reply from 172.16.76.5: bytes=32 time<1ms TTL=127
Reply from 172.16.76.5: bytes=32 time<1ms TTL=127
Reply from 172.16.76.5: bytes=32 time<1ms TTL=127
Reply from 172.16.76.5: bytes=32 time<1ms TTL=127
Reply from 172.16.76.5: bytes=32 time<1ms TTL=127
Reply from 172.16.76.5: bytes=32 time<1ms TTL=127
Reply from 172.16.76.5: bytes=32 time<1ms TTL=127
Reply from 172.16.76.5: bytes=32 time<1ms TTL=127
Reply from 172.16.76.5: bytes=32 time<1ms TTL=127
Reply from 172.16.76.5: bytes=32 time<1ms TTL=127
Reply from 172.16.76.5: bytes=32 time<1ms TTL=127
Reply from 172.16.76.5: bytes=32 time<1ms TTL=127
Reply from 172.16.76.5: bytes=32 time<1ms TTL=127
Reply from 172.16.76.5: bytes=32 time<1ms TTL=127
Reply from 172.16.76.5: bytes=32 time<1ms TTL=127
Reply from 172.16.76.5: bytes=32 time<1ms TTL=127
Reply from 172.16.76.5: bytes=32 time<1ms TTL=127
Reply from 172.16.76.5: bytes=32 time<1ms TTL=127

and my exchange server runs the Sophos Antivirus Console as well.

Edited October 4, 2012 by nickbro

[Darylrese]

pinging is still 56 - 200ms which is worrying. how much memory is your store using up? How much memory does your machine have allocated?

 

Will check on the NIC setup, a company come in and installed it for us

[Ergo]

@Darylrese In my experience, looking at the RAM usage of exchange servers can be very misleading when looking at performance problems. I would recommend looking at the overall performance details of the server including disk IO, CPU usage, NIC traffic before you start chasing a solution. Memory usage would only generally cause performance problems if it results in significant page file usage (ie disk IO) which would (in my experience) rarely be the case on Exchange 2010 servers.

 

For a virtual server you should be able to see some of the performance metrics from the virtualisation layer (VMware, Hyper-V, XenServer) which can be a more accurate view than the performance counters inside Windows but generally either will point you quickly in the right direction. Also worth noting that your ping speeds and other network speed issued could be completely separate from the servers (eg. a saturated network link between network cabinets/switches). To test for that specifically try pinging from different locations (eg. your PC, server sharing the same switch as exchange, etc).

 

Regards,

 

David

[Darylrese]

Thank-you. The PC's that are pinging the server are going directly into our core switch with a gigabyte link, so there shouldnt be an issue!

 

Il try and get some screen shots of the VMWare console performance tab for the exchange

[Darylrese]

Here a bunch of screens on performance from the VMWare console for exchange:

 

[Domino]

as before, the exchange store will use all the RAM it can, don't necessarily take that as an issue.

 

CPU could be an issue though, MS recommend 4x processor cores for a virtual MBX role, and the same for CAS and Transport. as you're running combined you may not need 8, but have you got the capacity to increase on what you've got there?

[Ergo]

Looks from your email like something funny is happening with your networking - it looks like there is a 10Mbps limit on network throughput which the server is constantly hitting against. I cannot see from the config screenshot which VMware NIC you have specified, but I would assume E1000. Need to ensure your VMware tools are installed and up to date within the VM, and may be worth looking at the physical network throughput against the host in case there is a wider problem.

 

I have been advised by one of my colleages to look at whether you have traffic shaping set on the VM which could be limiting the server's access to network bandwidth. Also the network traffic does look like it is a "new" pattern may be worth looking to see if you can identify the source of that traffic (is it a client machine, other server, etc).

 

Regards,

 

David

Edited October 4, 2012 by Ergo

[Darylrese]

Domino - Im sure we could allocate two more vCPU cores if it would help!

 

 

Speaking to the company who put it in, they still think its a memory issue, im not convinced. A backup of our exchange has so far taken 22hours and backups were only setup earlier this week which is when we started having issues. I think the backup may be causing all the network traffic!

 

[DMcCoy]

Pings that high aren't going to be memory or CPU related. The network is obviously maxed out at 10Mb by the backup process (which is why it's taking so long).

 

Why are you using the E1000 instead of the vmxnet adapter? I assume there are no limits set on the NIC in vmware?

[Ergo]

Thanks for the additional detail. I would say that from the sounds of it the backup is the likely cause of your problems. I would suggest your only way to really be sure would be to stop the backup (possibly not ideal as it may have to be re-started from the begining if you do). I assume it is trying to complete a full backup of the exchange databases across the network as your initial backup?

 

If you don't mind me asking what backup software/method are you using?

 

Regards,

 

David

[Darylrese]

We use Backup Exc 12 to backup to our SAN and then every friday onto tape, however it was only implimented on monday which is when it started to choke. They gave us no training on VMWare which is why im a little new to it, sorry guys!

 

Not sure why we are using E1000 instead for vmxnet for above reason, it was just setup like that from the company. Whats the difference?

[DMcCoy]

We use Backup Exc 12 to backup to our SAN and then every friday onto tape, however it was only implimented on monday which is when it started to choke. They gave us no training on VMWare which is why im a little new to it, sorry guys!

 

Not sure why we are using E1000 instead for vmxnet for above reason, it was just setup like that from the company. Whats the difference?

 

It may be depending on the defaults used by vmware are the time VMware KB: Choosing a network adapter for your virtual machine for a little info about them. My 2008R2 is using the vmxnet3 as the default.

 

Even the E1000 should be fine though, check the resource allocation along with the actual NIC speed on the host settings in vshpere to make sure it's running at 1Gb

[Ergo]

There are many reasons why your backups may be slow, and while they are running causing performance problems but I would in the first instance get a good undersatanding of how your VMware storage and backups are configured. There are many articles available online about how best to set up Backup Exec 2012 to back up VMware (including applications/databases within VMs) such as:

 

BE2012 - Design/strategy - VMware based with GRT OR regular backups with periodic VMware backups | Symantec Connect Community

 

As you have recently procured the solution I would recommend working with the supplier in the first instance to resolve - as this appears to be a backup issue I would hope they prioritise the issue for you. To ensure that we are not recommending you look at the wrong thing I would suggest testing performance/pings, etc when the backup is not running - I would expect the results to be significantly different if the backups are the cause.

 

Regards,

 

David

[Domino]

Pings that high aren't going to be memory or CPU related.

 

You'd be surprised, the exchange services have a higher priority than ping responses, so they can blat everything else out of the picture.

 

Anyway, I'd agree it looks like a network config issue - the CPUs could still do with a life to recommended levels anyway.

 

You've two nics on different vlans it appears - which is on your network, and how are the nics setup in windows?

[Darylrese]

You're right we do seem to have 2 NIC's on different VLAN's. Im guessing because we are part of the academy's forest aswell. This is whats in windows

 

[Domino]

I'd speak to the company that set it up and ask if there's any reason for not using the vmxnet3 card type - if not, ensure the vmware tools are up to date and change the network adapters for VMXNET3 ones.

 

are their other vms on the host showing similar symptoms? might be worth checking their isn't a speed/duplex mismatch between the host and the switch also - or that one isn't hardcoded to a slower speed.

 

But also as ergo said, trying it without the backup running is also worthwhile.

[DMcCoy]

Where are the actual physical hosts and virtual hosts in relation to each other? If the backup is using all of a 10Mb WAN link could explain it.

[Darylrese]

Backup was stopped and exchange restarted.... suprise suprise back to <1ms pings and outlook working perfectly. I have a feeling as soon as backup runs again its going to kill it. They did allocate 16GB to exchange too.

 

What should the network card speed be set to? Can we just add another network card VMXNET3 copy settings across (ip, subnet etc) and remove the E1000 cards?

Edited October 4, 2012 by Darylrese

[Domino]

Backup was stopped and exchange restarted.... suprise suprise back to <1ms pings and outlook working perfectly. I have a feeling as soon as backup runs again its going to kill it. They did allocate 16GB to exchange too.

 

What should the network card speed be set to? Can we just add another network card VMXNET3 copy settings across (ip, subnet etc) and remove the E1000 cards?

 

Yeah, the card switch should be pretty easy, as above.

 

The card speed should be autonegotiate - unless there's a pressing reason for it not to be

[Darylrese]

Backup rerun, only took an hour this time and exchange still working as it should be. Network useage now virtually nothing compared to maxed out as before.

 

Will look into adding another 2 vCPU's and the VMXNET3 cards to help speed things up in future. Ping is <1ms or =1ms now which is fine and roll up 7 for service pack 1 has been installed!

 

Cheers for your help guys!

[Ergo]

Cheers for your help guys!

 

Glad you got it sorted

 

Dave