HP DL360p Gen8 - Crashes since the microcode updates

[mrbios]

I can't remember when it was, Sometime between January and April there were some windows updates that led to my backup server falling over once every 1-2 days, it'd need a hard reboot to get it to come back again each time.

 

There was then a bios update relating to the spectre/meltdown fix(es) and ever since applying that the server now only falls over once every 1-2 weeks.

 

Has anyone encountered these problems and have a suggested solution? (please don't say rollback windows updates, i killed it doing that previously )

[kmount]

Do you have memory dumps enabled to have a chance to analyse what is causing it to crash?

 

(if not, enable them so they can be analysed to figure out what's happening)

 

Event log saying anything useful?

[Primus]

I have two of these patched and up to date with their most recent BIOS updates on Server 2012 r2 and Server 2016 and both are fine.

[mrbios]

Do you have memory dumps enabled to have a chance to analyse what is causing it to crash?

 

(if not, enable them so they can be analysed to figure out what's happening)

 

Event log saying anything useful?

 

In terms of event logs, I get an event ID 27 source ixgbi with the message that network link is disconnected. Along with an event id 10400 source NDIS saying the network interface has begun resetting. I've checked for updated network driver and tried rolling that back. That's the last message i get in system or application logs, then the machine is completely unresponsive from direct input. I've now enabled memory dumps.

 

EDIT: The network adaptor with the above messages is an additional Intel X540-T1 adaptor, so maybe that's faulty. Also i lied, there was a driver update, i've run that, previous date was late 2016. New one is October 17. I'll see if that driver resolves the problem.

Edited June 8, 2018 by mrbios

[computer_expert]

Anything useful in the iLO logs? (event log and IML logs are the 2 you need)

[Arthur]

There was then a bios update relating to the spectre/meltdown fix(es) and ever since applying that the server now only falls over once every 1-2 weeks.

Which BIOS update do you currently have installed?

 

The latest or the update that was subsequently pulled by HP due to bugs?

[mrbios]

Which BIOS update do you currently have installed?

 

The latest or the update that was subsequently pulled by HP due to bugs?

 

It's the latest, was only applied 2-3 weeks ago.

[mrbios]

Not going to say for certain that i've fixed this yet....but i think i've fixed this at last.

 

Turns out it was my own stupid mistake. The backup server only has 1 x 10GB link, and is connected to a switch stack where everything else is connected to both. When setting everything else up with iSCSI i'd selected multipath, so i'd instinctively done this on the backup server, and it seems as though a combination of hosting the database on a different server + enabling multipath when it wasn't was occasionally triggering an additional path through the 1GB connection that wasn't intended for iSCSI. I think what was then happening is the path got smashed by the amount of load the backup put on it, while still trying to read and write to the database, which just caused everything to lock up and give me a network card + iSCSI PDU error. Moved the database to the same server and ensured multipath isn't ticked and a specific interface is selected on each iSCSI target and so far no crashing.