RabbieBurns Posted July 16, 2009 Posted July 16, 2009 One of my HDs in one of my servers seems to have packed in. I'm not sure yet if the BIOS picks it up yet, will do a reboot soon, but the disks definitely off-line. Would there be anything written to any of the system logs that I could check?
mac_shinobi Posted July 16, 2009 Posted July 16, 2009 (edited) One of my HDs in one of my servers seems to have packed in. I'm not sure yet if the BIOS picks it up yet, will do a reboot soon, but the disks definitely off-line. Would there be anything written to any of the system logs that I could check? Isnt there anything in event viewer ? When it was installed was there no diagnostic software installed? That will teach me not to read what section it is in Edited July 16, 2009 by mac_shinobi
powdarrmonkey Posted July 16, 2009 Posted July 16, 2009 His servers are all Gentoo, which is why it's in *nix If you've got SMART enabled, messages normally go to syslog if it's been set up. Same applies to RAID (do you use mdadm?). If it wasn't set up though, it's like saying 'who deleted this file' after the event, so your luck probably varies. 2
OutToLunch Posted July 16, 2009 Posted July 16, 2009 Depends on the brand of server/RAID device etc, but does it have an event log in the BIOS? Mine logs critical faults there, I would assume a disk borking is one of them 1
mac_shinobi Posted July 16, 2009 Posted July 16, 2009 His servers are all Gentoo, which is why it's in *nix If you've got SMART enabled, messages normally go to syslog if it's been set up. Same applies to RAID (do you use mdadm?). If it wasn't set up though, it's like saying 'who deleted this file' after the event, so your luck probably varies. I never read that section - doh !!
kmount Posted July 16, 2009 Posted July 16, 2009 If smartd is running then yes, you'll probably find some entries. Worth checking smartctl --all /dev/sdX When you say offline, has it vanished from /dev ? 1
RabbieBurns Posted July 16, 2009 Author Posted July 16, 2009 OK rebooted, when it tried to mount the disk it said it coudnt find a valid partition table. Checked /var/log/messages and it is riddled with errors: Jul 15 09:17:08 roberts hdf: status error: status=0x00 { } Jul 15 09:18:35 roberts hdf: status error: status=0x00 { } Jul 15 09:18:35 roberts hdf: status error: status=0x00 { } Jul 15 09:18:35 roberts end_request: I/O error, dev hdf, sector 65 Jul 15 09:18:35 roberts hdf: status error: status=0x00 { } Jul 16 13:07:11 roberts end_request: I/O error, dev hdf, sector 77 Jul 16 13:07:11 roberts hdf: status error: status=0x00 { } Jul 16 13:07:11 roberts hdf: status error: status=0x00 { } Jul 16 13:07:11 roberts end_request: I/O error, dev hdf, sector 79 Jul 16 13:07:11 roberts hdf: status error: status=0x00 { } Jul 16 13:07:11 roberts hdf: status error: status=0x00 { } Jul 16 13:07:11 roberts end_request: I/O error, dev hdf, sector 81 Jul 16 13:07:11 roberts hdf: status error: status=0x00 { } Jul 16 13:07:11 roberts hdf: status error: status=0x00 { } Jul 16 13:07:11 roberts end_request: I/O error, dev hdf, sector 83 Jul 16 13:07:11 roberts Buffer I/O error on device hdf1, logical block 10 Jul 16 13:07:11 roberts hdf: status error: status=0x00 { } Jul 16 13:07:11 roberts end_request: I/O error, dev hdf, sector 85 Jul 16 13:07:11 roberts hdf: status error: status=0x00 { } Jul 16 13:07:11 roberts hdf: status error: status=0x00 { } Jul 16 13:07:11 roberts end_request: I/O error, dev hdf, sector 87 Jul 16 13:07:11 roberts hdf: status error: status=0x00 { } Jul 16 13:07:11 roberts end_request: I/O error, dev hdf, sector 89 Jul 16 13:07:11 roberts hdf: status error: status=0x00 { } Jul 16 13:07:11 roberts hdf: status error: status=0x00 { } Jul 16 13:07:11 roberts hdf: status error: status=0x00 { } Jul 16 13:07:11 roberts end_request: I/O error, dev hdf, sector 91 Jul 16 13:07:11 roberts hdf: status error: status=0x00 { } Jul 16 13:07:11 roberts end_request: I/O error, dev hdf, sector 93 Jul 16 13:07:11 roberts hdf: status error: status=0x00 { } Jul 16 13:07:11 roberts hdf: status error: status=0x00 { } Jul 16 13:07:11 roberts hdf: status error: status=0x00 { } Jul 16 13:07:11 roberts end_request: I/O error, dev hdf, sector 63 Jul 16 13:07:11 roberts hdf: status error: status=0x00 { } Jul 16 13:07:11 roberts hdf: status error: status=0x00 { } Jul 16 13:07:11 roberts end_request: I/O error, dev hdf, sector 65 Jul 16 13:07:11 roberts hdf: status error: status=0x00 { } Jul 16 13:07:11 roberts hdf: status error: status=0x00 { } Jul 16 13:07:11 roberts end_request: I/O error, dev hdf, sector 67 Jul 16 13:07:11 roberts hdf: status error: status=0x00 { } ... truncated to fit fdisk wasnt even listing it before the reboot but It shows it now, albeit after a 10s pause when it gets to hdf, and it logged this Jul 16 14:21:44 roberts hdf: lost interrupt Jul 16 14:21:44 roberts hdf: task_in_intr: status=0x7f { DriveReady DeviceFault SeekComplete DataRequest CorrectedError Index Error } Jul 16 14:21:44 roberts hdf: task_in_intr: error=0x7f { DriveStatusError UncorrectableError SectorIdNotFound TrackZeroNotFound AddrMarkNotFound }, LBAsect=8832734988296, sector=1 Jul 16 14:21:44 roberts ide: failed opcode was: unknown Jul 16 14:21:45 roberts ide2: reset: success heres the smart output roberts ~ # smartctl -a /dev/hdf smartctl version 5.38 [i686-pc-linux-gnu] Copyright (C) 2002-8 Bruce Allen Home page is http://smartmontools.sourceforge.net/ === START OF INFORMATION SECTION === Model Family: Seagate Barracuda 7200.10 family Device Model: ST3500630A Serial Number: 9QG3GMAP Firmware Version: 3.AAF User Capacity: 500,107,862,016 bytes Device is: In smartctl database [for details use: -P show] ATA Version is: 7 ATA Standard is: Exact ATA specification draft version not indicated Local Time is: Thu Jul 16 14:31:36 2009 BST SMART support is: Available - device has SMART capability. SMART support is: Enabled === START OF READ SMART DATA SECTION === SMART overall-health self-assessment test result: PASSED See vendor-specific Attribute list for marginal Attributes. General SMART Values: Offline data collection status: (0x82) Offline data collection activity was completed without error. Auto Offline Data Collection: Enabled. Self-test execution status: ( 0) The previous self-test routine completed without error or no self-test has ever been run. Total time to complete Offline data collection: ( 430) seconds. Offline data collection capabilities: (0x5b) SMART execute Offline immediate. Auto Offline data collection on/off support. Suspend Offline collection upon new command. Offline surface scan supported. Self-test supported. No Conveyance Self-test supported. Selective Self-test supported. SMART capabilities: (0x0003) Saves SMART data before entering power-saving mode. Supports SMART auto save timer. Error logging capability: (0x01) Error logging supported. General Purpose Logging supported. Short self-test routine recommended polling time: ( 1) minutes. Extended self-test routine recommended polling time: ( 163) minutes. SMART Attributes Data Structure revision number: 10 Vendor Specific SMART Attributes with Thresholds: ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE 1 Raw_Read_Error_Rate 0x000f 114 087 006 Pre-fail Always - 71633148 3 Spin_Up_Time 0x0003 098 093 000 Pre-fail Always - 0 4 Start_Stop_Count 0x0032 100 100 020 Old_age Always - 243 5 Reallocated_Sector_Ct 0x0033 100 100 036 Pre-fail Always - 0 7 Seek_Error_Rate 0x000f 071 043 030 Pre-fail Always - 215497672802 9 Power_On_Hours 0x0032 085 085 000 Old_age Always - 14005 10 Spin_Retry_Count 0x0013 100 100 097 Pre-fail Always - 0 12 Power_Cycle_Count 0x0032 100 100 020 Old_age Always - 329 187 Reported_Uncorrect 0x0032 100 100 000 Old_age Always - 0 189 High_Fly_Writes 0x003a 100 100 000 Old_age Always - 0 190 Airflow_Temperature_Cel 0x0022 063 044 045 Old_age Always In_the_past 37 (Lifetime Min/Max 37/37) 194 Temperature_Celsius 0x0022 037 056 000 Old_age Always - 37 (0 13 0 0) 195 Hardware_ECC_Recovered 0x001a 109 052 000 Old_age Always - 23068426 197 Current_Pending_Sector 0x0012 100 100 000 Old_age Always - 0 198 Offline_Uncorrectable 0x0010 100 100 000 Old_age Offline - 0 199 UDMA_CRC_Error_Count 0x003e 200 200 000 Old_age Always - 0 200 Multi_Zone_Error_Rate 0x0000 100 253 000 Old_age Offline - 0 202 TA_Increase_Count 0x0032 100 253 000 Old_age Always - 0 SMART Error Log Version: 1 No Errors Logged SMART Self-test log structure revision number 1 Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error # 1 Short offline Completed without error 00% 8796 - SMART Selective self-test log data structure revision number 1 SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS 1 0 0 Not_testing 2 0 0 Not_testing 3 0 0 Not_testing 4 0 0 Not_testing 5 0 0 Not_testing Selective self-test flags (0x0): After scanning selected spans, do NOT read-scan remainder of disk. If Selective self-test is pending on power-up, resume after 0 minute delay. Sorry for the long post.. can anyone figure out whats going on here? Thanks
RabbieBurns Posted July 16, 2009 Author Posted July 16, 2009 This is a bit wierd, it just mounted fine there. /dev/hda though I think is the real problem, system drive, so I think that could be causing the problems? roberts# smartctl -a /dev/hda smartctl version 5.38 [i686-pc-linux-gnu] Copyright (C) 2002-8 Bruce Allen Home page is http://smartmontools.sourceforge.net/ === START OF INFORMATION SECTION === Model Family: Maxtor DiamondMax Plus 9 family Device Model: Maxtor 6Y250P0 Serial Number: Y63W9TJE Firmware Version: YAR41BW0 User Capacity: 251,000,193,024 bytes Device is: In smartctl database [for details use: -P show] ATA Version is: 7 ATA Standard is: ATA/ATAPI-7 T13 1532D revision 0 Local Time is: Thu Jul 16 15:04:17 2009 BST SMART support is: Available - device has SMART capability. SMART support is: Enabled === START OF READ SMART DATA SECTION === SMART overall-health self-assessment test result: PASSED See vendor-specific Attribute list for marginal Attributes. General SMART Values: Offline data collection status: (0x82) Offline data collection activity was completed without error. Auto Offline Data Collection: Enabled. Self-test execution status: ( 0) The previous self-test routine completed without error or no self-test has ever been run. Total time to complete Offline data collection: ( 363) seconds. Offline data collection capabilities: (0x5b) SMART execute Offline immediate. Auto Offline data collection on/off support. Suspend Offline collection upon new command. Offline surface scan supported. Self-test supported. No Conveyance Self-test supported. Selective Self-test supported. SMART capabilities: (0x0003) Saves SMART data before entering power-saving mode. Supports SMART auto save timer. Error logging capability: (0x01) Error logging supported. No General Purpose Logging support. Short self-test routine recommended polling time: ( 2) minutes. Extended self-test routine recommended polling time: ( 107) minutes. SMART Error Log Version: 1 ATA Error Count: 5455 (device log contains only the most recent five errors) CR = Command Register [HEX] FR = Features Register [HEX] SC = Sector Count Register [HEX] SN = Sector Number Register [HEX] CL = Cylinder Low Register [HEX] CH = Cylinder High Register [HEX] DH = Device/Head Register [HEX] DC = Device Command Register [HEX] ER = Error register [HEX] ST = Status register [HEX] Powered_Up_Time is measured from power on, and printed as DDd+hh:mm:SS.sss where DD=days, hh=hours, mm=minutes, SS=sec, and sss=millisec. It "wraps" after 49.710 days. Error 5455 occurred at disk power-on lifetime: 27335 hours (1138 days + 23 hours) When the command that caused the error occurred, the device was in an unknown state. After command completion occurred, registers were: ER ST SC SN CL CH DH -- -- -- -- -- -- -- 84 51 00 36 aa 17 e0 Error: ICRC, ABRT at LBA = 0x0017aa36 = 1550902 Commands leading to the command that caused the error were: CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name -- -- -- -- -- -- -- -- ---------------- -------------------- 25 00 10 36 aa 17 e0 08 00:08:49.360 READ DMA EXT 25 00 08 f6 c6 14 e0 08 00:08:49.360 READ DMA EXT 25 00 20 3e ae 17 e0 08 00:08:49.360 READ DMA EXT 25 00 18 26 ae 17 e0 08 00:08:49.360 READ DMA EXT 25 00 08 0e ae 17 e0 08 00:08:49.360 READ DMA EXT Error 5454 occurred at disk power-on lifetime: 27325 hours (1138 days + 13 hours) When the command that caused the error occurred, the device was in an unknown state. After command completion occurred, registers were: ER ST SC SN CL CH DH -- -- -- -- -- -- -- 84 51 00 00 00 00 f0 Error: ICRC, ABRT at LBA = 0x00000000 = 0 Commands leading to the command that caused the error were: CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name -- -- -- -- -- -- -- -- ---------------- -------------------- c8 00 01 00 00 00 f0 08 00:08:15.616 READ DMA c6 00 10 00 00 00 f0 08 00:08:15.600 SET MULTIPLE MODE 10 00 00 00 00 00 f0 08 00:08:15.600 RECALIBRATE [OBS-4] c8 00 01 00 00 00 f0 04 00:08:15.568 READ DMA c8 00 01 00 00 00 f0 08 00:08:15.360 READ DMA Error 5453 occurred at disk power-on lifetime: 27325 hours (1138 days + 13 hours) When the command that caused the error occurred, the device was in an unknown state. After command completion occurred, registers were: ER ST SC SN CL CH DH -- -- -- -- -- -- -- 84 51 00 00 00 00 f0 Error: ICRC, ABRT at LBA = 0x00000000 = 0 Commands leading to the command that caused the error were: CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name -- -- -- -- -- -- -- -- ---------------- -------------------- c8 00 01 00 00 00 f0 08 00:08:15.360 READ DMA c6 00 10 00 00 00 f0 08 00:08:15.328 SET MULTIPLE MODE 10 00 00 00 00 00 f0 08 00:08:15.312 RECALIBRATE [OBS-4] c8 00 01 00 00 00 f0 04 00:08:15.296 READ DMA c8 00 01 00 00 00 f0 00 00:08:15.088 READ DMA Error 5452 occurred at disk power-on lifetime: 27325 hours (1138 days + 13 hours) When the command that caused the error occurred, the device was in an unknown state. After command completion occurred, registers were: ER ST SC SN CL CH DH -- -- -- -- -- -- -- 84 51 00 00 00 00 f0 Error: ICRC, ABRT at LBA = 0x00000000 = 0 Commands leading to the command that caused the error were: CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name -- -- -- -- -- -- -- -- ---------------- -------------------- c8 00 01 00 00 00 f0 00 00:08:15.088 READ DMA b0 da 00 01 4f c2 b0 00 00:08:05.360 SMART RETURN STATUS 20 03 01 01 00 00 b0 00 00:08:05.312 READ SECTOR(S) ef 03 45 01 00 00 b0 00 00:08:05.312 SET FEATURES [set transfer mode] ef 03 0c 01 00 00 b0 00 00:08:05.248 SET FEATURES [set transfer mode] Error 5451 occurred at disk power-on lifetime: 27325 hours (1138 days + 13 hours) When the command that caused the error occurred, the device was in an unknown state. After command completion occurred, registers were: ER ST SC SN CL CH DH -- -- -- -- -- -- -- 84 51 00 00 00 00 f0 Error: ICRC, ABRT at LBA = 0x00000000 = 0 Commands leading to the command that caused the error were: CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name -- -- -- -- -- -- -- -- ---------------- -------------------- c8 00 01 00 00 00 f0 08 00:09:08.192 READ DMA c6 00 10 00 00 00 f0 08 00:09:08.176 SET MULTIPLE MODE 10 00 00 00 00 00 f0 08 00:09:08.176 RECALIBRATE [OBS-4] c8 00 01 00 00 00 f0 04 00:09:08.144 READ DMA c8 00 01 00 00 00 f0 08 00:09:07.920 READ DMA SMART Self-test log structure revision number 1 Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error # 1 Extended offline Completed without error 00% 32954 - # 2 Extended offline Completed without error 00% 32797 - # 3 Extended offline Completed without error 00% 32640 - # 4 Extended offline Completed without error 00% 32483 - # 5 Extended offline Completed without error 00% 32326 - # 6 Extended offline Completed without error 00% 32169 - # 7 Extended offline Completed without error 00% 32012 - # 8 Extended offline Completed without error 00% 31854 - # 9 Extended offline Completed without error 00% 31697 - #10 Extended offline Completed without error 00% 31540 - #11 Extended offline Completed without error 00% 31383 - #12 Extended offline Completed without error 00% 31226 - #13 Extended offline Completed without error 00% 31069 - #14 Extended offline Completed without error 00% 30912 - #15 Extended offline Completed without error 00% 30755 - #16 Extended offline Completed without error 00% 30610 - #17 Extended offline Completed without error 00% 30454 - #18 Extended offline Completed without error 00% 30296 - #19 Extended offline Completed without error 00% 30139 - #20 Extended offline Completed: read failure 20% 29981 343743310 #21 Extended offline Completed without error 00% 29825 - SMART Selective self-test log data structure revision number 1 SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS 1 0 0 Not_testing 2 0 0 Not_testing 3 0 0 Not_testing 4 0 0 Not_testing 5 0 0 Not_testing Selective self-test flags (0x0): After scanning selected spans, do NOT read-scan remainder of disk. If Selective self-test is pending on power-up, resume after 0 minute delay. What are normal hd / mobo temps? roberts / # sensors ... fan1: 2732 RPM (min = 0 RPM, div = 2) fan2: 0 RPM (min = 0 RPM, div = 2) fan3: 0 RPM (min = 0 RPM, div = 2) M/B Temp: +44 C (low = -1 C, high = +127 C) sensor = thermistor CPU Temp: +35 C (low = -1 C, high = +127 C) sensor = thermistor Temp3: -1 C (low = -1 C, high = +127 C) sensor = disabled
RabbieBurns Posted July 16, 2009 Author Posted July 16, 2009 Things seem to be running fine just now touch wood. Im a bit worried about the errors logged in the smart from hda right enough, but as its a 4 yr old maxtor I guess thats to be expected. Ive backed up everything off the system drive and made a stage 4, so it should be trivial to rebuild if/when hda packs in. Im not sure what caused the errors with hdf, but Ill keep an eye on it. Thanks for all the replies today.
RabbieBurns Posted July 17, 2009 Author Posted July 17, 2009 hmm so I try to login today and cant. Try to see whats up and see this http://www.peninsulafm.com/ts.png gr
kmount Posted July 17, 2009 Posted July 17, 2009 Looks bricked. Run fsck as it asks, least you have ipkvm to reinstall gentoo EDIT: ^ The is in sympathy.
RabbieBurns Posted July 17, 2009 Author Posted July 17, 2009 Looks bricked. Run fsck as it asks, least you have ipkvm to reinstall gentoo EDIT: ^ The is in sympathy. So you reckon its the actual system disk thats gubbed? It isnt accepting my root password or control d for some reason over the ipkvm. Will need to wait till someone can reboot it. Times like this im glad i set up dhcp failover etc accross both..
kmount Posted July 17, 2009 Posted July 17, 2009 Second looking at that it might be failing and dropping into maintenance mode because of hdf. I'd suggest getting your remote hands to remove that disk and boot it up and see how it is. I did recall earlier though you had issues with hda too, think it might not be a good disk time for you ahead..
RabbieBurns Posted July 17, 2009 Author Posted July 17, 2009 hda is a 250gb 5yr old maxtor rubbish, which has probably been powered on 24/7 the whole 5 years so its not suprising really, I guess it probably could do with a rebuild. The hdf on the other hand is a fairly new seagate 500gb, still under warranty though according to the site.
kmount Posted July 17, 2009 Posted July 17, 2009 The later extended test completed OK on hda so it might have been a blip. hdf seems to have these read errors, could maybe be controller/cable?
Geoff Posted July 17, 2009 Posted July 17, 2009 Have you run any of the manufacturers HDD testing tools on the various drives?
RabbieBurns Posted July 17, 2009 Author Posted July 17, 2009 Not since November or December. And I dont think seagate make a linux tool so I cant do it now. (dont have physical access to server and a bit too complicated to get the remote hands to do )
Recommended Posts
Create an account or sign in to comment
You need to be a member in order to leave a comment
Create an account
Sign up for a new account in our community. It's easy!
Register a new accountSign in
Already have an account? Sign in here.
Sign In Now