Moving VM's from datastores

[synaesthesia]

Scenario:

 

One ESXi host with 2 raid cards (identical) and a raid array on each.

 

One is the new one, one is the old one. It hosts a single guest which I need to move from one to the other.

 

Copying one of the virtual disks over (which is 1.2TB) is HORRENDOUSLY slow. As in, it's currently got 50 hours remaining.

These aren't particularly slow drives, RAID10 arrays on both.

That's just a standard copy operation from within the GUI. I don't particularly want to risk using vmkfstools in SSH to do it with the reports of it timing out for large files for so many people, plus I don't trust the thick/thin provisioning properties of it.

 

Is there a quicker way to do it? Starting to think restoring from backup would be far faster!

[DMcCoy]

Beyond VAAI, vCenter and shared storage? Not really. You may indeed find a windows server backup of a large data drive and restore is quicker, if you can find fast and large storage in the middle.

[synaesthesia]

That's something we don't have. Just ESXi and vSphere.

 

3157 minutes remaining Floppy disks weren't this slow.

[glennda]

Is it on shared storage?

[synaesthesia]

As the original post, it's dedicated, local storage and neither are loaded/doing anything.

 

Actually going up in time and has been fairly steadily. 4347 minutes remaining now. I wouldn't mind, but we rather need this working by tomorrow night, and I have another 800GB to copy over to it >:|

Edited April 8, 2014 by synaesthesia

[Marshall_IT]

My experience with esxi and local storage is always slow. I just don't think they designed it to work that way. Not much help to you right now but you certainly have my sympathy!

[kmount]

Use Veeam quick migrate (free version can do it too), tends to work out a lot quicker than a full fat copy and it'll work live most of the time except the suspend + boot up at the end.

[glennda]

As the original post, it's dedicated, local storage and neither are loaded/doing anything.

 

Actually going up in time and has been fairly steadily. 4347 minutes remaining now. I wouldn't mind, but we rather need this working by tomorrow night, and I have another 800GB to copy over to it >:|

 

Sorry i misread it!

[synaesthesia]

No worries @glennda. Cheers, @kmount - will that work when the VM is staying on the same host, just a different datastore? I imagine it's a similar process to using vcenter converter (which is my next line of enquiry!)

 

Something isn't right regardless - it managed to fail and in doing so lost the VMDK descriptor file. I excreted a brick thinking it'd corrupted the whole virtual disk, thankfully found a guide on recreating that file and the server is back up and running. I'm not convinced the new datastore is working right. It's pretty much identical to the old one, the drives are identical spec just twice the size, same RAID controller, same array type yet it's diabolically slow. Copying a 1GB ISO file to the old datastore takes 10 seconds. 3 minutes to the new one!

Edited April 8, 2014 by synaesthesia

[kmount]

Yes, it will work fine on the same host, just two different datastores. (did this last night to move a vCenter VM which was offline as I did not have storage vmotion to move it online).

 

I would not use the converter because it will re-mac etc. (though you could override it).

 

Using the quick migrate will allow you to continue running 'live' whilst it does its thing then it'll snapshot + copy differences at the end.

 

- - - Updated - - -

 

Block in VMware the same?

Block/strip size on the HBA the same?

[synaesthesia]

I'm going to check those when I'm on site in the morning. I've just run ESXTop out of interest...

 

Copying a 7GB ISO to the old datastore (which is prefixed naa.GUID for some reason) DQLEN 254 CMDS/S 1508.95 WRITES/s 1335.49 MBWRTN/s 217.62

 

Copying to the new one, the DS i'm trying to copy the above VM to... DQLEN 254 CMDS/s 119.90 WRITES/s 109.05 MBWRTN/s 21.88. That has a T10.guid prefix on that. I'm not sure on the relevance of those.

 

I suspect there's an issue there.

 

Both are VMFS5, block size in VMWare 1MB on both. I'll power off tomorrow morning and check the raid controller itself. That's quite a serious difference in performance. 4 x WD RE 2TB in RAID10. (The old datastore is 4 x WD RE 1TB in RAID10)

Edited April 8, 2014 by synaesthesia

[synaesthesia]

Sigh. Everything's identical. The only difference was firmware (the "new" raid controller had an older firmware). Upgraded that and it's made no difference at all. 14MB/s according to esxtop. Really not good.

[PotNoodleTech]

What is your status of the write cache on both arrays? I am suspecting the second (new) array does not have write cache enabled wheras maybe the first one does?

 

Give us more hardware specs too?

[tmcd35]

I can only offer two thoughts, possibilities - Is the writeback cache settings the same on both controllers? Do they have the same amount of cache ram?

 

Also, could it be a PCI bandwidth problem? What happens if you swap the cards over? or move the new one to another slot?

[synaesthesia]

All settings identical.

System is Intel 5500 motherboard with twin quad core xeon e5520, 20GB RAM.

Controller (new) is a 9650SE-12ML., 256MB cache (same on both, other was a 9650SE-4ML. New one is higher spec)

 

I've just put the old array on the new controller to see what performance that gets.#

 

****

 

Controller isn't right. Swapped controllers, drives fine.

Edited April 9, 2014 by synaesthesia

[synaesthesia]

Turns out I must have gone about the original array a bit shonky. Started from scratch, doing exactly as I believed I had done before and things are considerably more speedy!

 

Currently using vCenter converter to shift the VM across, it's going along rather better. Only getting 30-40MB/s at the moment but that's as it's grabbing from an iScsi datastore about as far away from everything as possible. Hits 100 from the other array thankfully.

 

Cheers for all the input