One of my HDs in one of my servers seems to have packed in. I'm not sure yet if the BIOS picks it up yet, will do a reboot soon, but the disks definitely off-line. Would there be anything written to any of the system logs that I could check?
Printable View
One of my HDs in one of my servers seems to have packed in. I'm not sure yet if the BIOS picks it up yet, will do a reboot soon, but the disks definitely off-line. Would there be anything written to any of the system logs that I could check?
Isnt there anything in event viewer ?Quote:
Originally Posted by RabbieBurns
When it was installed was there no diagnostic software installed?
That will teach me not to read what section it is in
His servers are all Gentoo, which is why it's in *nix :p
If you've got SMART enabled, messages normally go to syslog if it's been set up. Same applies to RAID (do you use mdadm?). If it wasn't set up though, it's like saying 'who deleted this file' after the event, so your luck probably varies.
Depends on the brand of server/RAID device etc, but does it have an event log in the BIOS? Mine logs critical faults there, I would assume a disk borking is one of them :)
I never read that section - doh !! :getmecoat:Quote:
Originally Posted by powdarrmonkey
If smartd is running then yes, you'll probably find some entries.
Worth checking smartctl --all /dev/sdX
When you say offline, has it vanished from /dev ?
OK rebooted, when it tried to mount the disk it said it coudnt find a valid partition table. Checked /var/log/messages and it is riddled with errors:
fdisk wasnt even listing it before the reboot but It shows it now, albeit after a 10s pause when it gets to hdf, and it logged thisCode:Jul 15 09:17:08 roberts hdf: status error: status=0x00 { }
Jul 15 09:18:35 roberts hdf: status error: status=0x00 { }
Jul 15 09:18:35 roberts hdf: status error: status=0x00 { }
Jul 15 09:18:35 roberts end_request: I/O error, dev hdf, sector 65
Jul 15 09:18:35 roberts hdf: status error: status=0x00 { }
Jul 16 13:07:11 roberts end_request: I/O error, dev hdf, sector 77
Jul 16 13:07:11 roberts hdf: status error: status=0x00 { }
Jul 16 13:07:11 roberts hdf: status error: status=0x00 { }
Jul 16 13:07:11 roberts end_request: I/O error, dev hdf, sector 79
Jul 16 13:07:11 roberts hdf: status error: status=0x00 { }
Jul 16 13:07:11 roberts hdf: status error: status=0x00 { }
Jul 16 13:07:11 roberts end_request: I/O error, dev hdf, sector 81
Jul 16 13:07:11 roberts hdf: status error: status=0x00 { }
Jul 16 13:07:11 roberts hdf: status error: status=0x00 { }
Jul 16 13:07:11 roberts end_request: I/O error, dev hdf, sector 83
Jul 16 13:07:11 roberts Buffer I/O error on device hdf1, logical block 10
Jul 16 13:07:11 roberts hdf: status error: status=0x00 { }
Jul 16 13:07:11 roberts end_request: I/O error, dev hdf, sector 85
Jul 16 13:07:11 roberts hdf: status error: status=0x00 { }
Jul 16 13:07:11 roberts hdf: status error: status=0x00 { }
Jul 16 13:07:11 roberts end_request: I/O error, dev hdf, sector 87
Jul 16 13:07:11 roberts hdf: status error: status=0x00 { }
Jul 16 13:07:11 roberts end_request: I/O error, dev hdf, sector 89
Jul 16 13:07:11 roberts hdf: status error: status=0x00 { }
Jul 16 13:07:11 roberts hdf: status error: status=0x00 { }
Jul 16 13:07:11 roberts hdf: status error: status=0x00 { }
Jul 16 13:07:11 roberts end_request: I/O error, dev hdf, sector 91
Jul 16 13:07:11 roberts hdf: status error: status=0x00 { }
Jul 16 13:07:11 roberts end_request: I/O error, dev hdf, sector 93
Jul 16 13:07:11 roberts hdf: status error: status=0x00 { }
Jul 16 13:07:11 roberts hdf: status error: status=0x00 { }
Jul 16 13:07:11 roberts hdf: status error: status=0x00 { }
Jul 16 13:07:11 roberts end_request: I/O error, dev hdf, sector 63
Jul 16 13:07:11 roberts hdf: status error: status=0x00 { }
Jul 16 13:07:11 roberts hdf: status error: status=0x00 { }
Jul 16 13:07:11 roberts end_request: I/O error, dev hdf, sector 65
Jul 16 13:07:11 roberts hdf: status error: status=0x00 { }
Jul 16 13:07:11 roberts hdf: status error: status=0x00 { }
Jul 16 13:07:11 roberts end_request: I/O error, dev hdf, sector 67
Jul 16 13:07:11 roberts hdf: status error: status=0x00 { }
... truncated to fit
heres the smart outputCode:Jul 16 14:21:44 roberts hdf: lost interrupt
Jul 16 14:21:44 roberts hdf: task_in_intr: status=0x7f { DriveReady DeviceFault SeekComplete DataRequest CorrectedError Index Error }
Jul 16 14:21:44 roberts hdf: task_in_intr: error=0x7f { DriveStatusError UncorrectableError SectorIdNotFound TrackZeroNotFound AddrMarkNotFound }, LBAsect=8832734988296, sector=1
Jul 16 14:21:44 roberts ide: failed opcode was: unknown
Jul 16 14:21:45 roberts ide2: reset: success
Sorry for the long post.. can anyone figure out whats going on here?Code:roberts ~ # smartctl -a /dev/hdf
smartctl version 5.38 [i686-pc-linux-gnu] Copyright (C) 2002-8 Bruce Allen
Home page is http://smartmontools.sourceforge.net/
=== START OF INFORMATION SECTION ===
Model Family: Seagate Barracuda 7200.10 family
Device Model: ST3500630A
Serial Number: 9QG3GMAP
Firmware Version: 3.AAF
User Capacity: 500,107,862,016 bytes
Device is: In smartctl database [for details use: -P show]
ATA Version is: 7
ATA Standard is: Exact ATA specification draft version not indicated
Local Time is: Thu Jul 16 14:31:36 2009 BST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
See vendor-specific Attribute list for marginal Attributes.
General SMART Values:
Offline data collection status: (0x82) Offline data collection activity
was completed without error.
Auto Offline Data Collection: Enabled.
Self-test execution status: ( 0) The previous self-test routine completed
without error or no self-test has ever
been run.
Total time to complete Offline
data collection: ( 430) seconds.
Offline data collection
capabilities: (0x5b) SMART execute Offline immediate.
Auto Offline data collection on/off support.
Suspend Offline collection upon new
command.
Offline surface scan supported.
Self-test supported.
No Conveyance Self-test supported.
Selective Self-test supported.
SMART capabilities: (0x0003) Saves SMART data before entering
power-saving mode.
Supports SMART auto save timer.
Error logging capability: (0x01) Error logging supported.
General Purpose Logging supported.
Short self-test routine
recommended polling time: ( 1) minutes.
Extended self-test routine
recommended polling time: ( 163) minutes.
SMART Attributes Data Structure revision number: 10
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate 0x000f 114 087 006 Pre-fail Always - 71633148
3 Spin_Up_Time 0x0003 098 093 000 Pre-fail Always - 0
4 Start_Stop_Count 0x0032 100 100 020 Old_age Always - 243
5 Reallocated_Sector_Ct 0x0033 100 100 036 Pre-fail Always - 0
7 Seek_Error_Rate 0x000f 071 043 030 Pre-fail Always - 215497672802
9 Power_On_Hours 0x0032 085 085 000 Old_age Always - 14005
10 Spin_Retry_Count 0x0013 100 100 097 Pre-fail Always - 0
12 Power_Cycle_Count 0x0032 100 100 020 Old_age Always - 329
187 Reported_Uncorrect 0x0032 100 100 000 Old_age Always - 0
189 High_Fly_Writes 0x003a 100 100 000 Old_age Always - 0
190 Airflow_Temperature_Cel 0x0022 063 044 045 Old_age Always In_the_past 37 (Lifetime Min/Max 37/37)
194 Temperature_Celsius 0x0022 037 056 000 Old_age Always - 37 (0 13 0 0)
195 Hardware_ECC_Recovered 0x001a 109 052 000 Old_age Always - 23068426
197 Current_Pending_Sector 0x0012 100 100 000 Old_age Always - 0
198 Offline_Uncorrectable 0x0010 100 100 000 Old_age Offline - 0
199 UDMA_CRC_Error_Count 0x003e 200 200 000 Old_age Always - 0
200 Multi_Zone_Error_Rate 0x0000 100 253 000 Old_age Offline - 0
202 TA_Increase_Count 0x0032 100 253 000 Old_age Always - 0
SMART Error Log Version: 1
No Errors Logged
SMART Self-test log structure revision number 1
Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error
# 1 Short offline Completed without error 00% 8796 -
SMART Selective self-test log data structure revision number 1
SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS
1 0 0 Not_testing
2 0 0 Not_testing
3 0 0 Not_testing
4 0 0 Not_testing
5 0 0 Not_testing
Selective self-test flags (0x0):
After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.
Thanks :)
Temps look a bit high?
Ben
This is a bit wierd, it just mounted fine there.
/dev/hda though I think is the real problem, system drive, so I think that could be causing the problems?
What are normal hd / mobo temps?Code:roberts# smartctl -a /dev/hda
smartctl version 5.38 [i686-pc-linux-gnu] Copyright (C) 2002-8 Bruce Allen
Home page is http://smartmontools.sourceforge.net/
=== START OF INFORMATION SECTION ===
Model Family: Maxtor DiamondMax Plus 9 family
Device Model: Maxtor 6Y250P0
Serial Number: Y63W9TJE
Firmware Version: YAR41BW0
User Capacity: 251,000,193,024 bytes
Device is: In smartctl database [for details use: -P show]
ATA Version is: 7
ATA Standard is: ATA/ATAPI-7 T13 1532D revision 0
Local Time is: Thu Jul 16 15:04:17 2009 BST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
See vendor-specific Attribute list for marginal Attributes.
General SMART Values:
Offline data collection status: (0x82) Offline data collection activity
was completed without error.
Auto Offline Data Collection: Enabled.
Self-test execution status: ( 0) The previous self-test routine completed
without error or no self-test has ever
been run.
Total time to complete Offline
data collection: ( 363) seconds.
Offline data collection
capabilities: (0x5b) SMART execute Offline immediate.
Auto Offline data collection on/off support.
Suspend Offline collection upon new
command.
Offline surface scan supported.
Self-test supported.
No Conveyance Self-test supported.
Selective Self-test supported.
SMART capabilities: (0x0003) Saves SMART data before entering
power-saving mode.
Supports SMART auto save timer.
Error logging capability: (0x01) Error logging supported.
No General Purpose Logging support.
Short self-test routine
recommended polling time: ( 2) minutes.
Extended self-test routine
recommended polling time: ( 107) minutes.
SMART Error Log Version: 1
ATA Error Count: 5455 (device log contains only the most recent five errors)
CR = Command Register [HEX]
FR = Features Register [HEX]
SC = Sector Count Register [HEX]
SN = Sector Number Register [HEX]
CL = Cylinder Low Register [HEX]
CH = Cylinder High Register [HEX]
DH = Device/Head Register [HEX]
DC = Device Command Register [HEX]
ER = Error register [HEX]
ST = Status register [HEX]
Powered_Up_Time is measured from power on, and printed as
DDd+hh:mm:SS.sss where DD=days, hh=hours, mm=minutes,
SS=sec, and sss=millisec. It "wraps" after 49.710 days.
Error 5455 occurred at disk power-on lifetime: 27335 hours (1138 days + 23 hours)
When the command that caused the error occurred, the device was in an unknown state.
After command completion occurred, registers were:
ER ST SC SN CL CH DH
-- -- -- -- -- -- --
84 51 00 36 aa 17 e0 Error: ICRC, ABRT at LBA = 0x0017aa36 = 1550902
Commands leading to the command that caused the error were:
CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name
-- -- -- -- -- -- -- -- ---------------- --------------------
25 00 10 36 aa 17 e0 08 00:08:49.360 READ DMA EXT
25 00 08 f6 c6 14 e0 08 00:08:49.360 READ DMA EXT
25 00 20 3e ae 17 e0 08 00:08:49.360 READ DMA EXT
25 00 18 26 ae 17 e0 08 00:08:49.360 READ DMA EXT
25 00 08 0e ae 17 e0 08 00:08:49.360 READ DMA EXT
Error 5454 occurred at disk power-on lifetime: 27325 hours (1138 days + 13 hours)
When the command that caused the error occurred, the device was in an unknown state.
After command completion occurred, registers were:
ER ST SC SN CL CH DH
-- -- -- -- -- -- --
84 51 00 00 00 00 f0 Error: ICRC, ABRT at LBA = 0x00000000 = 0
Commands leading to the command that caused the error were:
CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name
-- -- -- -- -- -- -- -- ---------------- --------------------
c8 00 01 00 00 00 f0 08 00:08:15.616 READ DMA
c6 00 10 00 00 00 f0 08 00:08:15.600 SET MULTIPLE MODE
10 00 00 00 00 00 f0 08 00:08:15.600 RECALIBRATE [OBS-4]
c8 00 01 00 00 00 f0 04 00:08:15.568 READ DMA
c8 00 01 00 00 00 f0 08 00:08:15.360 READ DMA
Error 5453 occurred at disk power-on lifetime: 27325 hours (1138 days + 13 hours)
When the command that caused the error occurred, the device was in an unknown state.
After command completion occurred, registers were:
ER ST SC SN CL CH DH
-- -- -- -- -- -- --
84 51 00 00 00 00 f0 Error: ICRC, ABRT at LBA = 0x00000000 = 0
Commands leading to the command that caused the error were:
CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name
-- -- -- -- -- -- -- -- ---------------- --------------------
c8 00 01 00 00 00 f0 08 00:08:15.360 READ DMA
c6 00 10 00 00 00 f0 08 00:08:15.328 SET MULTIPLE MODE
10 00 00 00 00 00 f0 08 00:08:15.312 RECALIBRATE [OBS-4]
c8 00 01 00 00 00 f0 04 00:08:15.296 READ DMA
c8 00 01 00 00 00 f0 00 00:08:15.088 READ DMA
Error 5452 occurred at disk power-on lifetime: 27325 hours (1138 days + 13 hours)
When the command that caused the error occurred, the device was in an unknown state.
After command completion occurred, registers were:
ER ST SC SN CL CH DH
-- -- -- -- -- -- --
84 51 00 00 00 00 f0 Error: ICRC, ABRT at LBA = 0x00000000 = 0
Commands leading to the command that caused the error were:
CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name
-- -- -- -- -- -- -- -- ---------------- --------------------
c8 00 01 00 00 00 f0 00 00:08:15.088 READ DMA
b0 da 00 01 4f c2 b0 00 00:08:05.360 SMART RETURN STATUS
20 03 01 01 00 00 b0 00 00:08:05.312 READ SECTOR(S)
ef 03 45 01 00 00 b0 00 00:08:05.312 SET FEATURES [Set transfer mode]
ef 03 0c 01 00 00 b0 00 00:08:05.248 SET FEATURES [Set transfer mode]
Error 5451 occurred at disk power-on lifetime: 27325 hours (1138 days + 13 hours)
When the command that caused the error occurred, the device was in an unknown state.
After command completion occurred, registers were:
ER ST SC SN CL CH DH
-- -- -- -- -- -- --
84 51 00 00 00 00 f0 Error: ICRC, ABRT at LBA = 0x00000000 = 0
Commands leading to the command that caused the error were:
CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name
-- -- -- -- -- -- -- -- ---------------- --------------------
c8 00 01 00 00 00 f0 08 00:09:08.192 READ DMA
c6 00 10 00 00 00 f0 08 00:09:08.176 SET MULTIPLE MODE
10 00 00 00 00 00 f0 08 00:09:08.176 RECALIBRATE [OBS-4]
c8 00 01 00 00 00 f0 04 00:09:08.144 READ DMA
c8 00 01 00 00 00 f0 08 00:09:07.920 READ DMA
SMART Self-test log structure revision number 1
Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error
# 1 Extended offline Completed without error 00% 32954 -
# 2 Extended offline Completed without error 00% 32797 -
# 3 Extended offline Completed without error 00% 32640 -
# 4 Extended offline Completed without error 00% 32483 -
# 5 Extended offline Completed without error 00% 32326 -
# 6 Extended offline Completed without error 00% 32169 -
# 7 Extended offline Completed without error 00% 32012 -
# 8 Extended offline Completed without error 00% 31854 -
# 9 Extended offline Completed without error 00% 31697 -
#10 Extended offline Completed without error 00% 31540 -
#11 Extended offline Completed without error 00% 31383 -
#12 Extended offline Completed without error 00% 31226 -
#13 Extended offline Completed without error 00% 31069 -
#14 Extended offline Completed without error 00% 30912 -
#15 Extended offline Completed without error 00% 30755 -
#16 Extended offline Completed without error 00% 30610 -
#17 Extended offline Completed without error 00% 30454 -
#18 Extended offline Completed without error 00% 30296 -
#19 Extended offline Completed without error 00% 30139 -
#20 Extended offline Completed: read failure 20% 29981 343743310
#21 Extended offline Completed without error 00% 29825 -
SMART Selective self-test log data structure revision number 1
SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS
1 0 0 Not_testing
2 0 0 Not_testing
3 0 0 Not_testing
4 0 0 Not_testing
5 0 0 Not_testing
Selective self-test flags (0x0):
After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.
Code:roberts / # sensors
...
fan1: 2732 RPM (min = 0 RPM, div = 2)
fan2: 0 RPM (min = 0 RPM, div = 2)
fan3: 0 RPM (min = 0 RPM, div = 2)
M/B Temp: +44 C (low = -1 C, high = +127 C) sensor = thermistor
CPU Temp: +35 C (low = -1 C, high = +127 C) sensor = thermistor
Temp3: -1 C (low = -1 C, high = +127 C) sensor = disabled
Things seem to be running fine just now touch wood. Im a bit worried about the errors logged in the smart from hda right enough, but as its a 4 yr old maxtor I guess thats to be expected. Ive backed up everything off the system drive and made a stage 4, so it should be trivial to rebuild if/when hda packs in. Im not sure what caused the errors with hdf, but Ill keep an eye on it.
Thanks for all the replies today.
hmm so I try to login today and cant. Try to see whats up and see this
http://www.peninsulafm.com/ts.png
gr
Looks bricked.
Run fsck as it asks, least you have ipkvm to reinstall gentoo :)
EDIT: ^ The :) is in sympathy.
So you reckon its the actual system disk thats gubbed?Quote:
Originally Posted by kmount
It isnt accepting my root password or control d for some reason over the ipkvm. Will need to wait till someone can reboot it.
Times like this im glad i set up dhcp failover etc accross both..
Second looking at that it might be failing and dropping into maintenance mode because of hdf. I'd suggest getting your remote hands to remove that disk and boot it up and see how it is.
I did recall earlier though you had issues with hda too, think it might not be a good disk time for you ahead..
hda is a 250gb 5yr old maxtor rubbish, which has probably been powered on 24/7 the whole 5 years so its not suprising really, I guess it probably could do with a rebuild. The hdf on the other hand is a fairly new seagate 500gb, still under warranty though according to the site.