Server 2008 R2 locking up

[andyt57]

One of the schools that I support is out in the sticks and often has power dropouts. It goes through UPS batteries at a great rate. Looking in the logs the server ran throughout the summer but a power outage caused it to fall over in the first week in September. Ever since then I have had problems with the server, which logs up under load (i.e. when a full class try to log in on their laptops). It has locked up 20 times over the last month.

 

The server is a Dell PowerEdge T620 with an Intel Xeon E5-2637 @ 3GHz and 16Gb RAM. The C: drive is a 265Gb SSD and the data is on a RAID array running on a Dell PERC H310 RAID controller. All the drives are good. The are about 90 client devices, all running Windows 7 Professional 32-bit.

 

I first tried making sure that all updates were installed on the server and ran chkdsk on the SSD which found not errors but that did not resolve the issue.

 

I then tried taking an image on the C: drive and restoring it onto another SSD. The server booted without any issues but continued to lockup under load.

 

My thoughts tended towards believing that the system outage had corrupted the 2008 R2 installation somehow so, since the school has a Server 2012 R2 licence, I decided to try an in place upgrade. This seemed to go smoothly until one of the server reboots into 2012 R2 (blue screen with circling dots) when it reported an error and that is was reverting to Server 2008 R2.

 

Back in Server 2008 R2 the log reported a number of "The driver detected a controller error on \Device\Harddisk2\DR2" and "The default transaction resource manager on volume Svr2012R2 encountered an error while starting and its metadata was reset. The data contains the error code." This lead me to believe that the server firmware needed updating so I downloaded the latest BIOS from the Dell site along with the latest PERC H310 driver for 2008 R2 Sp1 (SAS-RAID_Driver_CFHFJ_WN64_6.805.03.00_A09). The BIOS updated without any problems, but when I tried to update the PERC H310 drivers I got "This Update package is not compatible with your system configuration"!, even though it is compatible with PERC H310 and Server 2008 R2.

 

I am now at a loss. I am only at the school for 3 hours each week and the other problems I have to sort out don't leave me with much time to work on the server. I do not have time to rebuild the server during term time and I am loathe to install server 2012 R2 if it won't work.

 

Does anyone have any suggestions?

 

Andy

[TwistedHelixis]

I think the 'This Update package is not compatible with your system configuration' might be due to needing to update iDRAC separately first but I might be wrong.

 

Have you got open manage installed on the server and does that list any issues?

 

BTW the H310 raid controller cards are not a great choice

 

What I did in the past at a primary school when I was really stuck was purchase a HP Micro server (I wanted one anyway), stick a couple of 2tb HDD in mirrored mode in and install Windows 2012r2, I then migrated everything to this. It actually worked really well and gave me plenty of time to workout what was up with the faulty server. Since then I have always used Hyper-v with virtual machines which make moving servers around so much simpler.

[TwistedHelixis]

Also is the server under warranty? if so give dell a call and they will extract all the logs and let you know if its a hardware issue or not

[computer_expert]

You can upgrade the firmware of various bits and bobs on these servers by using the integrated lifecycle controller. I've run it on a few servers and it downloaded and installed the latest firmware from the dell website - only issue is the server will be offline for the duration of the upgrade progress.

 

https://www.dell.com/support/article/uk/en/ukbsdt1/sln292343/how-to-update-dell-server-using-the-integrated-lifecycle-controller-update-platform---lcc-update?lang=en

 

It may be worth running dell diagnostics too. again, the server will be offline for the duration of the tests. https://www.dell.com/support/article/uk/en/ukbsdt1/sln283546/how-to-run-hardware-diagnostics-on-your-poweredge-server?lang=en#2

 

edit:

 

Dell also have a firmware update ISO image here that you can use to update if you have issues with proxy servers etc. You may need to run the firmware update ISO several times depending on how out of date the firmware is

Edited November 2, 2017 by computer_expert

[andyt57]

Also is the server under warranty? if so give dell a call and they will extract all the logs and let you know if its a hardware issue or not

Unfortunately the warranty ran out in May. Sod's law.

 

Andy

[andyt57]

You can upgrade the firmware of various bits and bobs on these servers by using the integrated lifecycle controller. I've run it on a few servers and it downloaded and installed the latest firmware from the dell website - only issue is the server will be offline for the duration of the upgrade progress.

 

https://www.dell.com/support/article/uk/en/ukbsdt1/sln292343/how-to-update-dell-server-using-the-integrated-lifecycle-controller-update-platform---lcc-update?lang=en

 

It may be worth running dell diagnostics too. again, the server will be offline for the duration of the tests. https://www.dell.com/support/article/uk/en/ukbsdt1/sln283546/how-to-run-hardware-diagnostics-on-your-poweredge-server?lang=en#2

 

edit:

 

Dell also have a firmware update ISO image here that you can use to update if you have issues with proxy servers etc. You may need to run the firmware update ISO several times depending on how out of date the firmware is

 

Thanks very much. I'll download the ISO and try that next week. It's a Primary School so the children go home at 3 and I'm there until 4:30 so taking the server down is not a problem as long as the teachers are warned beforehand.

 

Andy

[andyt57]

I think the 'This Update package is not compatible with your system configuration' might be due to needing to update iDRAC separately first but I might be wrong.

 

Have you got open manage installed on the server and does that list any issues?

 

BTW the H310 raid controller cards are not a great choice

 

What I did in the past at a primary school when I was really stuck was purchase a HP Micro server (I wanted one anyway), stick a couple of 2tb HDD in mirrored mode in and install Windows 2012r2, I then migrated everything to this. It actually worked really well and gave me plenty of time to workout what was up with the faulty server. Since then I have always used Hyper-v with virtual machines which make moving servers around so much simpler.

 

That's not a bad idea. I already have a MicroServer at home running NAS4Free. The new Gen 10 boxes are over £300 from HP (with £50 back) but the Gen 8 boxes are a bit cheaper online and I have a couple of spare 3Tb Seagate drives which would do the trick.

 

Andy

[andyt57]

You can upgrade the firmware of various bits and bobs on these servers by using the integrated lifecycle controller. I've run it on a few servers and it downloaded and installed the latest firmware from the dell website - only issue is the server will be offline for the duration of the upgrade progress.

 

https://www.dell.com/support/article/uk/en/ukbsdt1/sln292343/how-to-update-dell-server-using-the-integrated-lifecycle-controller-update-platform---lcc-update?lang=en

 

It may be worth running dell diagnostics too. again, the server will be offline for the duration of the tests. https://www.dell.com/support/article/uk/en/ukbsdt1/sln283546/how-to-run-hardware-diagnostics-on-your-poweredge-server?lang=en#2

 

edit:

 

Dell also have a firmware update ISO image here that you can use to update if you have issues with proxy servers etc. You may need to run the firmware update ISO several times depending on how out of date the firmware is

 

 

I downloaded and ran the firmware update ISO, which took about 45 minutes, and it updated a number of things including the iPERC. I noticed that some updates didn't install, as they required prerequisites, so I'll try running it again next week.

I also booted into the lifecycle controller the system check gave an ePSA 4233.1 Failure, with error code 2000-0251 and validation 83529. According to the Dell site the ePSA 2000-0251 error message means 'Event Log - the log contains failing records'. The description states that 'this relates to BIOS events in servers only' and advises that I update the BIOS and repeat the diagnostics. But I have updated the BIOS and the firmware update ISO passed over the BIOS update stating that the BIOS was the latest version.

 

So I'll try it all again next week.

 

Andy

[TwistedHelixis]

Think you can also delete the log which will help but can't remember if you do that from open manage or lifecycle.

[AlanD]

I am now at a loss. I am only at the school for 3 hours each week and the other problems I have to sort out ...

Sounds as if a conversation is required with Headmaster - or School Manager so that they understand that 3 hours a week is not going to fix this In fact tiny bite sized chunks of time send repeatedly on a problem often wastes more time overall than spending consecutive days on a problem.

[andyt57]

Sounds as if a conversation is required with Headmaster - or School Manager so that they understand that 3 hours a week is not going to fix this In fact tiny bite sized chunks of time send repeatedly on a problem often wastes more time overall than spending consecutive days on a problem.

 

Don't get me started. The problem is that it's a small primary school with only about 145 pupils so they run hand to mouth. The school obtained a number of RM Minibooks from another school, but with only 2 power supplies. Using those power supplies I have created a Windows 7 image, installed it on all the devices and joined them to the network. But the school can't afford to buy the extra power supplies. So all this stuff from the government about the new funding formula is a load of cr*p. And the local council has just withdrawn all SEND funding for newly statemented children due to lack of money.

 

Andy

[AlanD]

Rm mini books ... do they still have 1GB memory...if so that’s going to make them painfully slow with win7 ...as will the slow HD unless you or previous school has updated them. And unless you configure them with a GPO or something they run at half speed or something when disconnected from a PSU. You can get PSU s for around £10 each of ebay. And pre AC wireless isn’t great. Maybe it would be better if they decided not to have ICT.

[andyt57]

Well, having run the lifecycle controller diagnostics and the Dell firmware update ISO I arrived at the school this afternoon to find that the server was even worse than ever, locking up and having to have a hard reboot 10 times during the last week.

 

Andy

[AlanD]

If you are doing a hard reboot (this is because the server is completely unresponsive..?) And its not because you have non enterprise disks in a RAID (which are failing - but don't allow the RAID controller to mark them as failed - because retries eventually get the data)? And you have ECC ram? Dual PSU?

 

Ideally you need to a check disk after any unexpected hard reset.....otherwise its not surprising it fails again....

 

...and it is the server ...its not the switch or network failing - and someone going along to push the server reset switch because its appearing not to be responsive?

 

And nothing in the event log prior to the "restart" that gives an indication of distress?

[andyt57]

If you are doing a hard reboot (this is because the server is completely unresponsive..?) And its not because you have non enterprise disks in a RAID (which are failing - but don't allow the RAID controller to mark them as failed - because retries eventually get the data)? And you have ECC ram? Dual PSU?

 

Ideally you need to a check disk after any unexpected hard reset.....otherwise its not surprising it fails again....

 

...and it is the server ...its not the switch or network failing - and someone going along to push the server reset switch because its appearing not to be responsive?

 

And nothing in the event log prior to the "restart" that gives an indication of distress?

Yes, it's a hard reboot using the power button as the server is totally non-responsive and users cannot log on. If you look at my first message I initially surface checked the 256Gb SSD the operating system is on but didn't find any faults and took an image which I restored on to a replacement SSD. The data is on a RAID 5 array of WD Black drives and they are all good. The server has 16Gb RAM, but I don't know what type without opening the box - which isn't easy as it's in a cabinet.

 

I have been through the logs and cannot find anything obvious before the report of power loss.

 

Well, I've now rerun the Dell deployment toolkit ISO, removed IIS and WSUS from the server and installed the latest updates but once again have run out of time. I'll try the in place upgrade for Server 2012 R2 next week.

 

Andy

[andyt57]

An update. The server froze and required a hard reboot about 9:30 every work day and sometimes again a couple of hours later over the last couple of weeks.

 

I set up the server so that when it next froze the staff could press the right Ctrl key and then the Scroll Lock key twice to produce a memory dump file. I have been through the file with WinDbg and can't see anything untoward.

 

So I have asked the staff to toggle the Caps Lock and Num Lock keys on the keyboard directly connected to the server to see if the lights still go on and off. That will hopefully point me in the direction of a hardware and software fault. Unfortunately it doesn't freeze when I'm actually in school.

Edited January 18, 2018 by andyt57

[AlanD]

Well, these faults rarely get better by themselves. I’d be swapping the RAM chips...or if there is plenty, just unplug one strap at a time....then change Ssd...

[Theldron]

Probably nothing to do with it, but thought I would put my two penneth in. Our file server kept freezing at random times and required a hard reset. Updated all firmware, new RAM everything, turned out to be the anti-virus software running constant scan on new files and caused it to freeze. Turned out we had standard ESET end-point anti-virus installed instead of ESET Server anti-virus. Edited January 22, 2018 by Theldron

[sted]

ive had similar sounding issues with dell servers. do you use iscsi for anything? after some windows updates it kept trying to reconnect with an iscsi device that no longer existed (but id forgotten to remove from the iscsi control panel) and over time the server got slower and slower requiring at one point a couple of reboots a day to keep it at a sensible speed mainly it just got slow i dont recall it needing hard power cycling but its been a while. probably not the same thing but easy to check for and rule out

[andyt57]

That's a good point; the school does use ESET. I didn't set up the server, so I'll check the next time that I'm in school. It won't be this week as I'm off to BETT.

 

Andy

 

Probably nothing to do with it, but thought I would put my two penneth in. Our file server kept freezing at random times and required a hard reset. Updated all firmware, new RAM everything, turned out to be the anti-virus software running constant scan on new files and caused it to freeze. Turned out we had standard ESET end-point anti-virus installed instead of ESET Server anti-virus.

[andyt57]

I have already changed the SSD. The server is inside a floor mounted soundproofed cabinet and it's an absolute pain to get at. I may have to start playing with the RAM if nothing else works just to eliminate it from the issue - if I can get at it.

 

Andy

 

 

Well, these faults rarely get better by themselves. I’d be swapping the RAM chips...or if there is plenty, just unplug one strap at a time....then change Ssd...

[eddyc]

Probably nothing to do with it, but thought I would put my two penneth in. Our file server kept freezing at random times and required a hard reset. Updated all firmware, new RAM everything, turned out to be the anti-virus software running constant scan on new files and caused it to freeze. Turned out we had standard ESET end-point anti-virus installed instead of ESET Server anti-virus.

 

Do you use dedupe? We had an issue where our file servers would lock up on the hour for upto 10 minutes at a time.

 

Turns out that dedupe was trying to run a background optimisation every hour in task scheduler.

 

I deleted all of the tasks between 7am and 7 pm and left all of the out of hours processes in.

 

The freezing then stopped.

 

When this used to happen, all the shares would disappear, mapped network drives would disconnect, open files would close, etc. effectively killing the network.

[andyt57]

Probably nothing to do with it, but thought I would put my two penneth in. Our file server kept freezing at random times and required a hard reset. Updated all firmware, new RAM everything, turned out to be the anti-virus software running constant scan on new files and caused it to freeze. Turned out we had standard ESET end-point anti-virus installed instead of ESET Server anti-virus.

Back at the school and the server is indeed running ESET Endpoint Antivirus 5.0.2225.0 which is the same as the clients are running. What should I be running on the server?

 

Andy

[Foresthippy]

Back at the school and the server is indeed running ESET Endpoint Antivirus 5.0.2225.0 which is the same as the clients are running. What should I be running on the server?

 

Andy

 

ESET File Security for Servers

[Theldron]

Back at the school and the server is indeed running ESET Endpoint Antivirus 5.0.2225.0 which is the same as the clients are running. What should I be running on the server?

 

Andy

 

Go to the ESET download page and download JDMackay said the file security for servers version. I had this for a while before i worked it out. Let us know how you get on.