Exchange running really slow

[Darylrese]

Hi

For the last 2 days or so our exchange has been grinding to a halt and outlook 2010 freezes and crashes, constantly displaying this message to users:

1.jpg

Pinging the server also seems worrying:

Untitled.jpg

Exchange is a virtual server with 12GB allocated memory. Its sitting at a constant 11GB useage and doesnt fluctuate atall. Is it bottlenecking? What else could be wrong?

Our old exchange used to only have 4GB memory and coped fine! We upgraded to new servers in the summer, coming away from an RM system. Its been working fine for the last 3 weeks but has just started happening this week.

Tried cache mode, still the same if not worse.

[Domino]

Exchange tends to grab as much RAM as it can, so don't worry about the ysage too much.

Is this a 2010 deployment? one server?

Has it got multiple network cards, and do you have them trunked/linked at all?

[Darylrese]

Yep Exchange 2010 and only one exchange server. Yes i believe there are two NIC's trunked together.

Our store.exe is taking up 11GB out of 12GB which i believe may be slowing the server down. Going to take it offline later, bump the server up to 16GB and limit the store.exe cache size to something like 12GB, leaving 4GB for other applications. Hopefully that will improve things?

[nickbro]

Exchange 2007+ is very RAM intensive, unlike 2003-. It may be that it doesn't like running as a virtual machine, but I can think why. The ping timings look very bad, does it had a dedicated NIC on the virtual server or is it sharing it with other servers?

Have you applied the latest Update Roll Ups? I think we're SP2 Roll Up 3 now.

My Exchange 2010 server is constant with it's ping:

Reply from 172.16.76.5: bytes=32 time<1ms TTL=127
Reply from 172.16.76.5: bytes=32 time<1ms TTL=127
Reply from 172.16.76.5: bytes=32 time<1ms TTL=127
Reply from 172.16.76.5: bytes=32 time<1ms TTL=127
Reply from 172.16.76.5: bytes=32 time<1ms TTL=127
Reply from 172.16.76.5: bytes=32 time<1ms TTL=127
Reply from 172.16.76.5: bytes=32 time<1ms TTL=127
Reply from 172.16.76.5: bytes=32 time<1ms TTL=127
Reply from 172.16.76.5: bytes=32 time<1ms TTL=127
Reply from 172.16.76.5: bytes=32 time<1ms TTL=127
Reply from 172.16.76.5: bytes=32 time<1ms TTL=127
Reply from 172.16.76.5: bytes=32 time<1ms TTL=127
Reply from 172.16.76.5: bytes=32 time<1ms TTL=127
Reply from 172.16.76.5: bytes=32 time<1ms TTL=127
Reply from 172.16.76.5: bytes=32 time<1ms TTL=127
Reply from 172.16.76.5: bytes=32 time<1ms TTL=127
Reply from 172.16.76.5: bytes=32 time<1ms TTL=127
Reply from 172.16.76.5: bytes=32 time<1ms TTL=127

and my exchange server runs the Sophos Antivirus Console as well.

[Darylrese]

pinging is still 56 - 200ms which is worrying. how much memory is your store using up? How much memory does your machine have allocated?

Will check on the NIC setup, a company come in and installed it for us

[Ergo]

@Darylrese In my experience, looking at the RAM usage of exchange servers can be very misleading when looking at performance problems. I would recommend looking at the overall performance details of the server including disk IO, CPU usage, NIC traffic before you start chasing a solution. Memory usage would only generally cause performance problems if it results in significant page file usage (ie disk IO) which would (in my experience) rarely be the case on Exchange 2010 servers.

For a virtual server you should be able to see some of the performance metrics from the virtualisation layer (VMware, Hyper-V, XenServer) which can be a more accurate view than the performance counters inside Windows but generally either will point you quickly in the right direction. Also worth noting that your ping speeds and other network speed issued could be completely separate from the servers (eg. a saturated network link between network cabinets/switches). To test for that specifically try pinging from different locations (eg. your PC, server sharing the same switch as exchange, etc).

Regards,

David

[Darylrese]

Thank-you. The PC's that are pinging the server are going directly into our core switch with a gigabyte link, so there shouldnt be an issue!

Il try and get some screen shots of the VMWare console performance tab for the exchange

[Darylrese]

Here a bunch of screens on performance from the VMWare console for exchange:

disk.jpgmemory.jpgnetwork.jpguseage.jpg

[Domino]

as before, the exchange store will use all the RAM it can, don't necessarily take that as an issue.

CPU could be an issue though, MS recommend 4x processor cores for a virtual MBX role, and the same for CAS and Transport. as you're running combined you may not need 8, but have you got the capacity to increase on what you've got there?

[Ergo]

Looks from your email like something funny is happening with your networking - it looks like there is a 10Mbps limit on network throughput which the server is constantly hitting against. I cannot see from the config screenshot which VMware NIC you have specified, but I would assume E1000. Need to ensure your VMware tools are installed and up to date within the VM, and may be worth looking at the physical network throughput against the host in case there is a wider problem.

I have been advised by one of my colleages to look at whether you have traffic shaping set on the VM which could be limiting the server's access to network bandwidth. Also the network traffic does look like it is a "new" pattern may be worth looking to see if you can identify the source of that traffic (is it a client machine, other server, etc).

Regards,

David

[Darylrese]

Domino - Im sure we could allocate two more vCPU cores if it would help!


Speaking to the company who put it in, they still think its a memory issue, im not convinced. A backup of our exchange has so far taken 22hours and backups were only setup earlier this week which is when we started having issues. I think the backup may be causing all the network traffic!

2.jpg

[DMcCoy]

Pings that high aren't going to be memory or CPU related. The network is obviously maxed out at 10Mb by the backup process (which is why it's taking so long).

Why are you using the E1000 instead of the vmxnet adapter? I assume there are no limits set on the NIC in vmware?

[Ergo]

Thanks for the additional detail. I would say that from the sounds of it the backup is the likely cause of your problems. I would suggest your only way to really be sure would be to stop the backup (possibly not ideal as it may have to be re-started from the begining if you do). I assume it is trying to complete a full backup of the exchange databases across the network as your initial backup?

If you don't mind me asking what backup software/method are you using?

Regards,

David

[Darylrese]

We use Backup Exc 12 to backup to our SAN and then every friday onto tape, however it was only implimented on monday which is when it started to choke. They gave us no training on VMWare which is why im a little new to it, sorry guys!

Not sure why we are using E1000 instead for vmxnet for above reason, it was just setup like that from the company. Whats the difference?

[DMcCoy]

We use Backup Exc 12 to backup to our SAN and then every friday onto tape, however it was only implimented on monday which is when it started to choke. They gave us no training on VMWare which is why im a little new to it, sorry guys!

Not sure why we are using E1000 instead for vmxnet for above reason, it was just setup like that from the company. Whats the difference?

It may be depending on the defaults used by vmware are the time VMware KB: Choosing a network adapter for your virtual machine for a little info about them. My 2008R2 is using the vmxnet3 as the default.

Even the E1000 should be fine though, check the resource allocation along with the actual NIC speed on the host settings in vshpere to make sure it's running at 1Gb