Dell Poweredge T440 - Green Flashing LED on Drives

[timbo343]

Our CCTV server has 8 drives installed and 3 of the 8 (drives 4,5,7) have green flashing LEDs on the drives.

 

The other day one of the drives in the RAID failed (configured in a RAID 0) so the drive was replaced and we had to rebuild the RAID, however it appears that we cannot add drives 4,5,7 to the RAID.

 

Does anyone know what would be causing these drives to have a flashing green LED against the drives?

 

I've looked online and Dell articles state:

 

Blinks green two times per second --- Identifying drive or preparing for removal

Off --- Drive ready for insertion or removal

The drive status indicator remains off until all hard drivers are initialized after the system is turned on. Drivers are not ready for insertion or removal during this time.

Blinks green, amber, and off --- Predicted drive failure

Blinks amber four times per second --- Drive failed

Blinks green slowly --- Drive rebuilding

Steady green Drive --- Online

Blinks green three seconds, amber three seconds, and off six seconds --- Rebuild aborted

 

Does this mean the drive is about to fail?

 

Thing is we have moved the drives around and it appears it's the bay that is a fault rather than the drive as when we move the drive from position 5 to position 2, the working drive from position 2 gives a flashing green LED.

[DavR]

First thing I'd do, is install the Dell Server Administrator on the server. That will be able to tell you what the status of the array and all the drives is/are.

 

Alternatively, if you've got the iDrac configured on the motherboard, that will be able to provide a GUI with a lot of that information on too. It's a lot easier than trying to interpret light info.

 

Do you really mean RAID0 in the config? Cos one disk failure in RAID0 would be the end of the whole array, as there's no fault tolerance.

[Koldov]

First thing I'd do, is install the Dell Server Administrator on the server. That will be able to tell you what the status of the array and all the drives is/are.

 

Alternatively, if you've got the iDrac configured on the motherboard, that will be able to provide a GUI with a lot of that information on too. It's a lot easier than trying to interpret light info.

 

Do you really mean RAID0 in the config? Cos one disk failure in RAID0 would be the end of the whole array, as there's no fault tolerance.

 

This...

 

I didn't see in your OP if any of the choices of green flashing lights matched with what you're actually seeing. Out of the choices:

 

Blinks green two times per second --- Identifying drive or preparing for removal

 

or

 

Blinks green slowly --- Drive rebuilding

 

A disk failure in RAID0 (striped) would normally mean the whole array is trashed iirc as there's no redundancy (parity), so no rebuild, no inserting new disks, or swapping them around. I think you'd have to go in and delete the array first, then start from scratch by creating a new one. Was it a brand new disk, can't remember exactly, but some arrays write RAID info onto the disk.

 

You should also be able to see the drive status in the RAID Controller setup during the boot sequence if you hit the right key combo (don't know off hand).

[timbo343]

First thing I'd do, is install the Dell Server Administrator on the server. That will be able to tell you what the status of the array and all the drives is/are.

 

Alternatively, if you've got the iDrac configured on the motherboard, that will be able to provide a GUI with a lot of that information on too. It's a lot easier than trying to interpret light info.

 

Do you really mean RAID0 in the config? Cos one disk failure in RAID0 would be the end of the whole array, as there's no fault tolerance.

 

Yup, it was set up as RAID0. The CCTV server has 7x 10Tb drives in it. The drive in position 7 had failed so took out the RAID. This is how the previous CCTV company set the server up.

 

I've installed the Server Administrator on the Server and it doesn't really display a right lot under Storage. When i jump into iDRAC to rebuild the RAID, i cannot see drives in position 4, 5, 7.

Edited September 12, 2024 by timbo343

[DavR]

Sounds like the array has had it, then. I'm assuming that the OS is running either on a separate array, or, an SD card.

 

I would start by re-seating drives 4,5,7 to see if you can bring them back online (are yours hot swap drives or not? If not a reboot may be required to detect them). There may be a force-online option in the GUI somewhere. If you can't, you may be looking at replacing those drives too.

 

Then it would be a case of deleting the array, and creating a new one from scratch. Go for RAID5 at the very least, you'll lose some of the capacity, but, you won't lose the whole array next time something fails.

 

You may also find there are important firmware updates for the controller and disks on the Dell support site. Sometimes this has been known to save disks presenting as failed or failing, might be worth a shot.

[ToyHeartsFan]

I doubt that they are SAS drives if they are 10TB each and I don't think SATA supports the same level of diagnostics. If you can't see the drives in iDrac they are probably toast but you could try pulling one and sticking it in a USB caddy to see if Windows detects it in disk management before you throw it in the recycling bin. Do you need 70TB of storage, thats a lot of cam footage you have lost which can't be recovered because it was in RAID 0.

[timbo343]

Sounds like the array has had it, then. I'm assuming that the OS is running either on a separate array, or, an SD card.

 

I would start by re-seating drives 4,5,7 to see if you can bring them back online (are yours hot swap drives or not? If not a reboot may be required to detect them). There may be a force-online option in the GUI somewhere. If you can't, you may be looking at replacing those drives too.

 

Then it would be a case of deleting the array, and creating a new one from scratch. Go for RAID5 at the very least, you'll lose some of the capacity, but, you won't lose the whole array next time something fails.

 

You may also find there are important firmware updates for the controller and disks on the Dell support site. Sometimes this has been known to save disks presenting as failed or failing, might be worth a shot.

Tried all the obvious points, even moved the drives around and still position 4,5,7 dont work.

 

The OS is on a 600gb drive in position 0 so that's safe on a different array.

 

I've been into the F10 configuration post-boot and built the array from there. That's where you tell it what drives to use and position 4,5,7 are not listed. Even after 6 hours of using half of the drives in a RAID0 setup, drives at 4,5,7 are still flashing.

Edited September 12, 2024 by timbo343

[timbo343]

I doubt that they are SAS drives if they are 10TB each and I don't think SATA supports the same level of diagnostics. If you can't see the drives in iDrac they are probably toast but you could try pulling one and sticking it in a USB caddy to see if Windows detects it in disk management before you throw it in the recycling bin. Do you need 70TB of storage, thats a lot of cam footage you have lost which can't be recovered because it was in RAID 0.

They arent SAS drives, they are the Seagate SkyHawk drives.

 

The drives arent toast as they work in other positions. One was toast as when we tried to rebuild the array the rebuild failed.

 

We have 60 Hikvision IP cameras on the system and the requirement is to keep footage for 60 days.

Edited September 12, 2024 by timbo343

[DavR]

If the drive positions themselves are toast, then you're looking at a warranty call to replace the drive backplane. If it's not in warranty, then, ouch. Have you tried running all the available firmwares - iDRAC, backplane, raid card?

 

The politest way I can think of describing that RAID setup is...... your CCTV guys have a very different approach to server configs than an IT installer would take. I'd be looking at the OS on two small, fast (ideally SSD), disks in RAID1, then the data array in big SATA disks on RAID5.

[DavR]

I've been into the F10 configuration post-boot and built the array from there. That's where you tell it what drives to use and position 4,5,7 are not listed. Even after 6 hours of using half of the drives in a RAID0 setup, drives at 4,5,7 are still flashing.

 

If position 4,5,7 are not available to add to the RAID config, that usually means they're still in use in another array.

 

Are you sure you've deleted the old array from the config first? Haven't done this in a long while, but I think may need to commit the delete on the old array, before you can start creating the new one.

[ToyHeartsFan]

You can't rebuild a RAID 0 array, RAID 0 means theres Zero redundancy.

 

You also can't just move drives between bays in an active array, the controller stores details of the virtual disk config on each drive in case the controller fails so that the controller can be replaced and the "foreign config" can be restored.

 

One of the pages I pulled up said you can't hot swap drives when they are configured in RAID 0 and you have to reboot the server after any changes, so you should have probably used idrac to delete the failed drive then shut the server down before trying to replace it and make a new array and virtual disk.

 

 

 

I'm guessing its an 8 bay server and the OS is on 1 drive with no redundancy and the other 7 bays were used for the 10TB sata data drives in Raid 0?

 

In which case you need to rebuild / reconfigure the server, the OS needs to be in raid 1 as a minimum you can then use the remaining 6 bays in raid 5, 6 or 10 for your CCTV storage.

 

I dont know if your raid controller will let you convert a standalone drive in to a RAID 1 pair but you will probably need to either reinstall the OS or restore it from a backup (if you have one).

 

 

As others have said if the bays are faulty you need to contact Dell there's nothing anyone on here can do to help you; if the array config has been messed up by the failed drive and you then moving drives around you should be able to delete the virtual disks / array but might need to initialize / erase each drive using idrac but I can't tell you the exact steps from memory alone.

 

You could always pull the lot stick them in a USB dock and erase them, delete all the configs on the raid controller, put the drives back in and start again.

 

 

https://www.dell.com/support/kbdoc/en-us/000131039/poweredge-tutorials-physical-disks-and-raid-controller-perc-on-servers#disk

 

https://www.dell.com/support/kbdoc/en-us/000131039/poweredge-tutorials-physical-disks-and-raid-controller-perc-on-servers#perc

 

 

 

 

You will also need to speak to your DPO and tell them your CCTV system has failed and you have lost 60 days of CCTV because it was incorrectly configured, they will then have to decide whether it is necessary to inform the ICO.

 

The DPO should also review the retention period because 60 days seems excessive and you have to be able to justify keeping the footage that long, once you have configured the CCTV array to use raid 5, 6 or 10 and the OS to use RAID 1 you will have less storage anyway.

[timbo343]

If position 4,5,7 are not available to add to the RAID config, that usually means they're still in use in another array.

 

Are you sure you've deleted the old array from the config first? Haven't done this in a long while, but I think may need to commit the delete on the old array, before you can start creating the new one.

When i first went to look at the server to re-create the RAID, position 4 was unselectable, so i could choose 1,2,3,5,6,7. Then after a couple of restarts of the server to make sure I knew which positions were which, 5 was unselectable, then after another restart 7 was unselectable.

 

Thr CCTV really isn't my problem, it's te site team's issue, they ask us to take a look and advise first. There is no specific hardware support contract on that server because the company that put it in were charging far too much per year for maintenance so it was pulled.

 

The server is i think less thann3 years old. My team were kept out of the loop during install - let's say it was a difficult period within support teams when it was installed but now they want our help.

 

When i re-created the temporary array with 4 disks, there was no array to delete.

 

There's been no setup notes, no config notes, had to guess most passwords / variations of passwords. I tell you, it's like walking through thick fog at times!

[DavR]

Sounds like it!

 

That drive behaviour sounds pretty weird, and does point towards the drive backplane or multiple drives being a bit iffy. I'd be wary of setting up anything new on that, if different drive positions were liable to drop in and out during a reboot.

 

Do you know what the warranty status is on that server? If it's under three years, might still be warrantied... but then, with how every else has been done with that server, it's hard to guess. I'd be putting in a warranty call about now, if I could.

 

Other options could be, take each disk out and verify it with drive testing software on a PC. Then either recommend replacement disks / backplane. Or, salvage the disks and put them in a NAS box.

[PotNoodleTech]

Erm - if it's Raid 0 then there is no drive swaping or rebuilding the array - if one drive fails you lose data and the array is compromised.

 

Id be expexting chunks of that stored CCTV data to be missing to be honest. How it's still running is a miracle!