talksr Posted November 11, 2015 Posted November 11, 2015 Hi, Have never had an issue like this, and would greatly appreciate any help on what could be going wrong. We have a brand new HP ProLiant ML350 Gen9 E5-2603v3 which is a matter of months old. On almost a daily basis now, when I log into it (Server 2012) it pops up with an unexpected shutdown box. At first, I thought the UPS must have been failing, so I removed this from the server. But still the messages continued. Next I wondered if there was a fault with the power in the server room, so I changed power sockets, but still this issue is persisting. I am in the process of ordering a new UPS for it, but I am beginning to wonder if there is actually something wrong with the server its self. It has two x HP 500W Flex Slot Platinum Hot Plug Power Supply Kit. When the issue began, I removed one of the power supply kits and ran it on just one directly to the mains, but this has not helped. Does anyone have any suggestions for what could be causing this? These are the errors I am seeing very regularly....
PotNoodleTech Posted November 11, 2015 Posted November 11, 2015 If you've ruled out problems with the actual power supply then it looks like some kind of hardware fault on the server itself. What software have you installed on it, if any? Are automatic windows updates enabled on it? 1
Geoff Posted November 11, 2015 Posted November 11, 2015 (edited) Excluding the PSU or a software issue (You did tell windows not to reboot rather to bluescreen instead right?). Test the ram. If it's not the ram, it's probably the motherboard. Edited November 11, 2015 by Geoff
talksr Posted November 11, 2015 Author Posted November 11, 2015 If you've ruled out problems with the actual power supply then it looks like some kind of hardware fault on the server itself. What software have you installed on it, if any? Are automatic windows updates enabled on it? It has Server 2012 R2 which seems to be running very well with no faults. - - - Updated - - - Excluding the PSU or a software issue (You did tell windows not to reboot rather to bluescreen instead). Test the ram. If it's not the ram, it's probably the motherboard. What did you mean by You did tell windows not to reboot rather to bluescreen instead? Do these HP servers have a ram test utility?
Geoff Posted November 11, 2015 Posted November 11, 2015 What did you mean by ? System Properties -> Startup & Recovery -> System Failure -> untick 'Automatically Restart' Do these HP servers have a ram test utility? No Idea, I usually just stick a memtest86+ disk in and leave it to it. 1
talksr Posted November 11, 2015 Author Posted November 11, 2015 System Properties -> Startup & Recovery -> System Failure -> untick 'Automatically Restart' No Idea, I usually just stick a memtest86+ disk in and leave it to it. Thanks, never used Memtest, will give it a go. - - - Updated - - - Presumably, if this is a motherboard fault, switching the MB out with another will not affect my OS or array? I have not had such a fault in a very long time, so a little unclear on any potential problems it could cause.
Geoff Posted November 11, 2015 Posted November 11, 2015 Are you using onboard hardware raid (rather than a separate card)? If so you must make sure the firmware versions match otherwise the array is liable to be unreadable. An easy way to ensure this as you have a semi-working system is to flash your firmware to the current latest version available (and hope the new motherboard doesn't come with an unreleased newer version).
witch Posted November 11, 2015 Posted November 11, 2015 If it is that new then it is under warranty. Go straight to HP
talksr Posted November 11, 2015 Author Posted November 11, 2015 Are you using onboard hardware raid (rather than a separate card)? If so you must make sure the firmware versions match otherwise the array is liable to be unreadable. An easy way to ensure this as you have a semi-working system is to flash your firmware to the current latest version available (and hope the new motherboard doesn't come with an unreleased newer version). No, I have a separate card on this one. It has one HP Smart Array P440/4GB FBWC 12Gb 1-port Int SAS Controller.
talksr Posted November 11, 2015 Author Posted November 11, 2015 System Properties -> Startup & Recovery -> System Failure -> untick 'Automatically Restart' Going back to this...... seeing as 'Write an event to the system log' is ticked, I would expect to see something in the log if the system was failing. But all we are seeing is that it restarted unexpectedly.
Geoff Posted November 11, 2015 Posted November 11, 2015 Right, you should see event id 1001 / 1003 followed by a 6003 if the OS was crashing.
dblight Posted November 11, 2015 Posted November 11, 2015 What does the iLo say? Have a look at the logs in the HP System Management app. Any hardware issues will show also
talksr Posted November 11, 2015 Author Posted November 11, 2015 (edited) Right, you should see event id 1001 / 1003 followed by a 6003 if the OS was crashing. Ok, I have searched the entire System part of the event log for 1001, 1003 and 6003 and there is nothing at all. - - - Updated - - - What does the iLo say? Have a look at the logs in the HP System Management app. Any hardware issues will show also Can't get on to the iLo at the moment as I am connected remotely and not sure of the iLo details. It has the HP Storage Administrator installed, but can't see any iLo software. Would I be able to download anything to look at it now, without physically going there? I have an HP Lights-Out Configuration Utility, but that is it. Edited November 11, 2015 by talksr
Geoff Posted November 11, 2015 Posted November 11, 2015 iLO is a seperate ethernet port in the server that you have to plug in and confgure separately to your LAN (preferably on some sort of management VLAN away from your normal traffic) port connections. 1
talksr Posted November 11, 2015 Author Posted November 11, 2015 iLO is a seperate ethernet port in the server that you have to plug in and confgure separately to your LAN (preferably on some sort of management VLAN away from your normal traffic) port connections. I was thinking that would be the case. Ok no worries, I will be there tomorrow, will update the call once I get a chance to connect and see.
john Posted November 11, 2015 Posted November 11, 2015 What does the iLo say? Have a look at the logs in the HP System Management app. Any hardware issues will show also That is the best way in the HP stuff on it you have loads of HP tools it will tell you more in that no doubt and shows status of components etc... Also did you do the Firmware when you got it and install it using Intelligent Provisioning to ensure it has the right drivers? 1
talksr Posted November 12, 2015 Author Posted November 12, 2015 Ok, I have plugged in a network connection to the iLO adapter and now have access to iLO. Status>System Health is showing as Degraded. If I click on System Health, Subsystems and Devices shows that Storage is degraded. If I click on Storage, I see the following..... It says that the Cache Module Status is degraded, but everything else seems fine. What can I do to rectify this? On a side note, Power Supply Status is looking good, no errors in that section. I am in the process of going through the iLO event log to see what was going on at the time of the resets. Last night I left an unsaved text file open on the server, and I note that it did not restart. I did also do as Geoff suggested and unticked reboot in Startup and Recovery.
talksr Posted November 12, 2015 Author Posted November 12, 2015 That is the best way in the HP stuff on it you have loads of HP tools it will tell you more in that no doubt and shows status of components etc... Also did you do the Firmware when you got it and install it using Intelligent Provisioning to ensure it has the right drivers? Thanks for your post, yes a very useful feature of HP hardware. I did not set this one up, but it certainly looks like Intelligent Provisioning was used. There are no missing drivers and it looks to be a very sound install.
Geoff Posted November 12, 2015 Posted November 12, 2015 (edited) If you put the cache module in the wrong slot, it will overheat and fail (I've seen one hit 85C). Move the P420 to the other slot. Additionally some have bad capacitors. Edited November 12, 2015 by Geoff
talksr Posted November 12, 2015 Author Posted November 12, 2015 If you put the cache module in the wrong slot, it will overheat and fail (I've seen one hit 85C). Move the P420 to the other slot. Additionally some have bad capacitors. Thanks for your helpful post. This is the slot it is currently in. What would be the wrong slot? Also, it would not mess my RAID up if I move it to a new slot would it?
Geoff Posted November 12, 2015 Posted November 12, 2015 You need to move it down in the picture, that corner of the case is dead air. The other way of forcing air over the top side of the card is to remove the blanking panel in the top slot (ideally replacing it with a vented one. Allowing air out of it. However this can impact air flow in the rest of the chassis. Moving a PCI card about will not affect the raid array. 1
talksr Posted November 25, 2015 Author Posted November 25, 2015 Hi, Just to give you all an update on this.... After spending over four hours talking to HP support, they have established that the raid controller is faulty. They confirmed they would be sending me one on 26th November, but are now asking for more evidence that we own the hardware. They asked for this today (24th), and the replacement is due to come tomorrow. I am not on site until tomorrow and as if things could not become more stressful, the school have reported that the server is down. I have connected via VLAN, authenticated the iLO page and note that the Storage Status has now changed to Failed. I am crossing my fingers that firstly, the honour their promise and send me the part tomorrow as they had told me and secondly that the raid will not have been wiped out by this.
Recommended Posts
Create an account or sign in to comment
You need to be a member in order to leave a comment
Create an account
Sign up for a new account in our community. It's easy!
Register a new accountSign in
Already have an account? Sign in here.
Sign In Now