[Hyper-V 2008 R2] - VHD giving I/O error

[Gatt]

As the title says, I have a VHD file that I need to move to a new Physical Disk but cannot do so:

 

I can boot the VM up, snapshot it, delete & merge snapshots, power it down, etc

 

What I CANNOT do is:

 

1) use VEEAM to create a backup

2) Copy the VHD

3) Move the VHD

 

If I try any of the following, it will generate an IO error, and 8/10 times it crashes the entire Hyper-visor and all the VMs in it.

When this happens I have to physically power off the Host server and reboot each VM one at a time

 

For most of the time when the affected server doesn't cause an issue, but occasionally it does - especially during a backup - which I've now disabled!

 

The VM is running SQL Server and has 2 VHDs attached - thankfully the one with all my Databases is on another physical disk and is not affected. The affected disk is the OS disk. Whilst I could technically build a new OS, it will take a lot of time to rebuild the OS and reconfigure SQL Server and the associated systems (SharePoint & SCCM being the 2 main systems that could likely complain a fair bit!)

 

Is there anyway I can shift the contents of the VHD (system & OS partitions) to a new, clean VHD?

 

I know I can mount the VHD file in Win server 2012, but I cant do anything with the System Partition..

[Cache]

Not a virtualisation expert by any means, however when I needed to increase the C Drive of one of our 2003 VM's I just ran it through VMware Converter again, so if you want to try and clone the VM, you could try going through a P2V type process again within the VM?

 

The other thing is, have you tried something as simple a chkdsk on it? (Again, don't know the effect it would have so perhaps wait for more opinions)

[VeryPC_Ed]

Hi Gatt,

 

Does this happen if you try exporting the VM (shutdown, right click and export VM) and then importing it ?

 

Ed

[Gatt]

Hi Gatt,

 

Does this happen if you try exporting the VM (shutdown, right click and export VM) and then importing it ?

 

Ed

Yeah

 

As soon as I do anything other than start/stop/manage snapshots the whole system becomes unresponsive

 

I tried today to convert it to a fixed disk and same thing happened at about 35/40% so kinda resigning myself to the task of spinning up a new OS disk and re-installing SQL Server

[VeryPC_Ed]

As a last resort you could try Disk2Vhd if the VM is bootable and chkdsk indicates nothing wrong with the VM filesystem but it sounds like some underlying issue with the VHD/Storage which needs looking into.

 

The thing that worries me is how its taking out the hypervisor as everything should be segregated, I'd be inclined to take everything offline and chkdsk the storage for your vhd's to ensure it's not an filesystem issue - also check your raid controller to see if any of your discs are reporting errors etc.

 

 

All that being said, a clean build would probably be better the long run but I'd double check to see if you can find the cause of the issue.

 

Ed

[Gatt]

I'm thinking the same thing about the physical media and desperately trying to get VMs moved off it. New HDD on the cards!

There's no RAID per say as this is really a "Home Lab" so shoe-string budgets apply here (2 x 2TB SATA hard drives)

 

Going to try and collar a DBA at work tomorrow to see if I can save the SQL Config (DB's on another drive and are safe) and import into a new server.

 

I'd also like to know why it's killing the HV as it should just kill the VMMS/VMWP not the whole lot! But it gets to a point where none of the servers are usable and I cannot clean restart the host.

[DMcCoy]

If it's a damaged disk, it may not be actually crashing, but stuck on a few sector reads for a *very* long time. Although if it goes on long enough the other machines may bsod due to a disk timeout. Are there disk errors on the host in event log?

[Gatt]

No, other than the expected when the host finally croaked (when I killed the VMWP's)

[Jamo]

No, other than the expected when the host finally croaked (when I killed the VMWP's)

 

Run the P2V tool on it (V2V) (I'm guessing Hyper V has an eqiuvalent) and clone it?

[Gatt]

@Jamo - I doubt this is going to work as anything that tries to read the VHD file (Hyper-V - Exportor Convert VHD, Veeam, Move or Copy) fails as soon as the bad sector is hit.

 

The only option I have is to rebuild the OS Disk of the VM affected to a new location and recreate the SQL Server config.

 

This is where I need some help - How do I backup or migrate SQL Server?

It will essentially be a replacement OS Disk inside the same VM.

All my DB's are on a second VHD so wont be affected.

 

My concerns are any SPNs and all my SQL Logins, etc...

 

Is there a way of exporting the "config" so that I can import it again before I re-attach all the Databases?

[Gatt]

Think I've got it fixed now!

 

Still in testing but managed to clone the affected VHD using CloneZilla.

Server is now running on a freshly cloned VHD disk on a new drive

 

Just gonna let it settle for a while, then try and Veeam it again tonight.

[VeryPC_Ed]

Glad to hear you are making some progress !

 

Ed

[Gatt]

Thanks - hopefully I have

 

I'll still need to replace the disk at some point but for now the affected VM is now in a safer place

 

An upside is that its reduced the size of the VHD by about a third!

Downside is I have a 120GB Virtual Disk that I dare not delete - don't want another VM getting affected by the [possible] bad sector!

[Jamo]

Thanks - hopefully I have

 

I'll still need to replace the disk at some point but for now the affected VM is now in a safer place

 

An upside is that its reduced the size of the VHD by about a third!

Downside is I have a 120GB Virtual Disk that I dare not delete - don't want another VM getting affected by the [possible] bad sector!

 

 

I have had to to V2Vs before in similar circumstances, once because a machine got stuck on a ESX host and wouldn't migrate off even when turned off!!

 

Clone is always a last resort but a good one which often works!

[KK20]

What is the guest OS? Can you backup the guest from within and restore the guest to a new fresh VHD (or physical)?