Server Shares Freezing - cannot save or close work

[witch]

I don't know if this is the right place to put this as I don't know if it is a server problem but here goes..

The server is an HP Proliant running Server 2008R2

It has been pretty reliable up until now.

Randomly, every few days, someone will come and tell me that they can't save as everything has frozen. I have to restart the server to unfreeze it.

There is nothing that I can see in the logs that would explain it.

I thought it was when the load increased - eg when a classful of children were logging on together but it has just done it at lunchtime when minimal people are logged on. (three weeks running it it did it at 1.20pm on a Monday)

I have solved it once by switching the switches off and on again - but that could be the removal of the load on the server that made it all unfreeze.

Has anyone ANY ideas what I can look at?

The switches are unmanaged and I have only managed to get hold of two spares that I could swap out. I had targeted the switch with most of the IT suite connections on it but today's event might make that a bit pointless.

I am really stuck this time as I have absolutely no idea where to start.

[fiza]

@witch - Has the server itself frozen or is it they cant access the shares? Are you able to log onto the Server at the console?

[witch]

Sorry - should have said - yes, I can log in at the server so probably server freeze are the wrong words

[Griff]

This sounds familiar to a problem we had last year....we had to update the drivers for the network card on the server and that fixed the issue. But its worth mentioning we also had a lengthy remote session with an engineer who tried various other fixes some of which he didn't log with us but I believe the driver fix resolved the issue. I have put my notes below from when we had the problem hopefully they will be helpful

 

Serv1 following problems;

 

Network is spiking heavily and grinds to a halt we are getting completely disconnected on the client side the mapped drives just drop out and we cannot ping the main server, all other servers running fine

 

I restarted the server and when launching the DNS it told me that the server was unavailable and gives me a box to connect to DNS SERVER when I select 'this computer' it fails again, on restarting the DNS service the issue is fixed and the network begins to work again

 

I have pulled a log from our main switch just encase that is causing an issue which I doubt. We are also seeing a pattern of when the server goes down it seems to be every 2 days and between 12:00am and 7:00am. I am going to try and reboot the server tomorrow night to see if that will delay any problems on the Wednesday.

 

Rebooting the server the night before has delayed the issue the issue reappeared on Thursday

 

Engineer reviewed log from switch and can find no issues

 

Remote session with Partnership Engineer they have looked at the DNS advised that we update Ethernet drivers

 

Updated to the latest Ethernet drivers server has been running for 4 days solid no issues

Edited June 11, 2014 by Griff
When I say spiking heavily I was able to open resource monitor to view activity on the server

[pete]

Are the switches on a UPS?

 

What else happens at 13:20 on a Monday?

 

Do the affected switches share a distribution board / supply with something like the canteen or other high "surging power draw".

 

If this were on our site, I'd be looking at power first.

[teejay]

I would also be looking at the NIC drivers, there are some versions for HP servers that are really borked. Be careful when updating though, do some research to check if anyone is having problems with the version you are going to update to.

[Heebeejeebee]

I would also be looking at the NIC drivers, there are some versions for HP servers that are really borked. Be careful when updating though, do some research to check if anyone is having problems with the version you are going to update to.

 

That - and check the firmware versions as well.

 

HBJB

[SYNACK]

Proliant MD/DL3?0 G? depending on the model it may have an integrated log viewer which will tell you about most hardware things that are afoot.

 

I would also look at the drivers, possibly set up performance counters to log during the day for stuff like Read/Write on the NIC and the drives along with the queue on each.

 

I would also look at the AV and make sure it is not collapsing connections that it does not like. We had all sorts of fun with Nortons AV cutting off connection from machines it thought were infected rightly or wrongly.

 

 

Another thing to check is turning off TCP acceleration and offloading in the NIC driver, especially if you have landed intel network cards in there and double especially if it hosts VMs. Had all sorts of issues with shoddy intel drivers going nuts and dropping connections to VMs.

 

 

EDIT: Watch the firmware versions as people have said, there was a firmware update a while back that bricked onboard NICs in hp servers.

Edited June 11, 2014 by SYNACK

[witch]

We have had some external support in over half term and we looked at the HP info together - we updated the NIC drivers, making sure that they weren't a problem but couldnt find anything wrong. When it all goes wrong, and I reboot either the server or the switches - which aren't on a UPS BTW - it all just quietly comes back. Once the server is back up the shares re-establish themselves and away it all goes.

However, and I have only just remembered this - the reason I thought the server had frozen was when a whole class were trying to log on, all PCs got stuck on "user profile" I think.

So we have that issue alongside the shares dropping out. No red crosses though, just no ability to save a doc to the share or open one on the share.

I am very confused

At 13.20 on a Monday a whole class logs in in the IT suite. But they weren't doing it today and it still all went wrong.

Will check to see if they are intel drivers.No VMs thank goodness. I've never had an issue with ESET - where would I look please?

[witch]

Have checked the server logs - nothing. I have no idea if some users are unaffected as they aren't all logged on or working in their "My Documents" or the shared resources drive.

No changes in hardware. Only microsoft updates

Will try and ping on Fri - not there tomorrow

[pcstru]

If you can log onto the server, what do you see in terms of CPU, Memory and network bandwidth use? Is the storage in rude health (sometimes things failing to quite fail can cause bottlenecks) ? What else happens at 1:20pm (sounds like a good time for a shedfull of activity where I am - start of class, someone telling all the kids - "OK, copy the Lord of The Rings MKV that I ripped from blu ray to your local drive from the server" ).

[witch]

Everything looks fine according to me and the external support guy who is very good.

What happens at 1.20 is the class logs on - but as I said, it happened today during lunchtime when no more than 10 people, and probably less, were active on the network

[markwilfan]

Could be a dying disk controller. We had similar probs a few years ago

[SYNACK]

Eep. Eset. Lots of people like it, I'm not one of them after their updates hosed our exchange server twice in a row and caused all sorts of networking issues. Everyone else seems to like it though.

 

I would look for anything labeled intrusion detection or network activity checking, connection throttling etc. And disable all of it.

 

I have also had it kill the entire network stack on client machines and only release it with a removal and reinstall. Luckily as it works most of the time it may just be hitting a connections per minute throttle and locking it out.

 

There should be detection logs somewhere in the application on the server and a section for network related stuff, throttling should be listed there but don't rule it out even if its not.

 

They do change their engine behavior without warning so eset could easily have tweaked something in an update to wreck your day. They did it to me enough times.

[witch]

Hmmm I will investigate

But I have ESET at my other school with pretty much the same server, and have seen no issues of any sort.

 

I just don't know where to start really.

The disk controller question strikes a bit of a chord somehow - don't know why - how would I investigate that?

[markwilfan]

I'm assuming your drives are raided in some form? Might be worth monitoring disk activity, however if the server is locking up you'd expect to see this shoot right down. I'm pretty sure we had errors in one of the windows logs when our was failing. Might be worth checking the disks out too with the manufacturers utilities to see if any are going south. To be honest this problem plagued us for a few months as it was difficult to troubleshoot.

[markwilfan]

Just a thought is shadow copy running at 1.20?

[witch]

Yes, it is raided. No errors in windows logs.

We did have an error reported that related to one of the backup drives - the backup eventually failed but replacing the drive seemed to sort that out although we did still get the odd error report. Since we replaced the internal USB that has gone away though.

I will look at the manufacturers utilities but I think the external guy did that

[SYNACK]

The utils you want are the hp acu (array configuration utility) and the hp integrated log viewer which will show up way more info than event viewer as they are designed to isolate the OS from the issues. As to eset, a single infected or broken machine spewing packets could trip it. A wireshark of the network to check for a busted Nic spewing rubbish packets or a virus doing the same could easily kill it like that.

[witch]

No shadow copy

[witch]

The utils you want are the hp acu (array configuration utility) and the hp integrated log viewer which will show up way more info than event viewer as they are designed to isolate the OS from the issues. As to eset, a single infected or broken machine spewing packets could trip it. A wireshark of the network to check for a busted Nic spewing rubbish packets or a virus doing the same could easily kill it like that.

hp acu and hp integrated log viewer not showing any errors, sadly. Have swapped out a switch - the one with the server connection - so will just have to wait now and see if the error recurs to eliminate that switch

[Bananas]

I know it's obvious but i'll ask just in case. Power save setting on the NIC along with speed and duplex hard set rather than auto?

[Stuclark]

I think you need to take a long hard look at how this server is set up / what it's doing.

 

Is it the only server you've got; in which case is it doing AD, DNS, DHCP, GPO *plus* being a file, print & backup server? Is it hosting VMs?

 

Can you build another server to move some of the roles / move the file share to?

 

What's the network infastructure like - are you getting massive bandwidth useage which is killing the switches?

 

How many users are trying to access the server at the same time when it "freezes"?

 

If, when the server has "frozen" you can log into the console and check task manager etc. - if that all looks ok then the chances are the server *itself* isn't the issue. If while logged in you can get to other network resources (internet etc.) then the nic is at least passing traffic. If however any of these fail, you have an idea where to start...

 

I suspect it's a nic driver / firmware issue (as others have said) - have you got any other nics you can put in the server and use instead of the current one?

 

... as a side note; I'm suprised your external consultant hasn't suggested (or done) some of the above...

[witch]

I know it's obvious but i'll ask just in case. Power save setting on the NIC along with speed and duplex hard set rather than auto?

 

 

I don't know. Nothing has been changed. I will check

[witch]

I think you need to take a long hard look at how this server is set up / what it's doing.

 

Is it the only server you've got; in which case is it doing AD, DNS, DHCP, GPO *plus* being a file, print & backup server? Is it hosting VMs?...

It is the only server. Yes it is doing AD DNS DHCP GPO, file, print and backup. No VMs

 

Can you build another server to move some of the roles / move the file share to?...

 

I have no other servers

 

What's the network infastructure like - are you getting massive bandwidth useage which is killing the switches?

Not that I can see - what would you use to check

 

How many users are trying to access the server at the same time when it "freezes"?

It was happening when a class of 32 logged on or off but it has happened randomly a few times as well

 

If, when the server has "frozen" you can log into the console and check task manager etc. - if that all looks ok then the chances are the server *itself* isn't the issue. If while logged in you can get to other network resources (internet etc.) then the nic is at least passing traffic. If however any of these fail, you have an idea where to start...

I can access task manager and log on and off when the issue occurs

 

I suspect it's a nic driver / firmware issue (as others have said) - have you got any other nics you can put in the server and use instead of the current one?

No, I haven't. The NIC drivers have been updated and the server does seem to run better but that hasn't changed the issue, sadly

 

.

.. as a side note; I'm suprised your external consultant hasn't suggested (or done) some of the above...

He did