[fog] Tasks Slowing Down

[tubaplayer83]

Hello,

 

We have been using our FOG server for a few months now and it has been running great. Since OCT 5th all tasks that have been run have taken over 7 hours to complete, this includes image uploading, deploying images and wiping a computer. The tasks start out running over 4GB/min and gradually slow down to under 500mb/min

 

Does anyone have any idea what could be causing this?

 

Thank you for your help

Edited October 17, 2012 by ZeroHour

[Geoff]

Use ifconfig to check if there are any errors reported on the network interface. Use iostat to see if there are any I/O load issues

[tubaplayer83]

I ran ifconfig and there were no errors reported, when i ran iostat it came up with bin/sh: iostat: not found. I aplogize, I'm kind of learning all of this on the fly, did I do that wrong?

[tubaplayer83]

OK, I figured it out.

 

ifconfig

 

UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1

RX packets:5470 errors:0 dropped:0 overruns:0 frame:0

TX packets:526 errors:0 dropped:0 overruns:0 carrier:0

collisions:0 txqueuelen:1000

RX bytes:1000414 (1.0 MB) TX bytes:102863 (102.8 KB)

Interrupt:21 Memory:dffe0000-e0000000

 

 

lo Link encap:Local Loopback

inet addr:127.0.0.1 Mask:255.0.0.0

inet6 addr: ::1/128 Scope:Host

UP LOOPBACK RUNNING MTU:16436 Metric:1

RX packets:68 errors:0 dropped:0 overruns:0 frame:0

TX packets:68 errors:0 dropped:0 overruns:0 carrier:0

collisions:0 txqueuelen:0

RX bytes:5070 (5.0 KB) TX bytes:5070 (5.0 KB)

 

 

virbr0 Link encap:Ethernet HWaddr d2:cf:68:16:33:80

inet addr:192.168.122.1 Bcast:192.168.122.255 Mask:255.255.255.0

UP BROADCAST MULTICAST MTU:1500 Metric:1

RX packets:0 errors:0 dropped:0 overruns:0 frame:0

TX packets:0 errors:0 dropped:0 overruns:0 carrier:0

collisions:0 txqueuelen:0

RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)

 

iostat

 

root@FOGSERVER:~# iostat

Linux 3.2.0-32-generic (FOGSERVER) 10/17/2012 _x86_64_ (2 CPU)

 

 

avg-cpu: %user %nice %system %iowait %steal %idle

1.35 0.36 0.81 1.64 0.00 95.84

 

 

Device: tps kB_read/s kB_wrtn/s kB_read kB_wrtn

sda 13.81 473.00 76.95 550108 89492

sdb 0.16 0.66 0.00 764 0

md0 0.16 0.64 0.00 748 0

dm-0 24.26 470.66 76.94 547393 89488

dm-1 0.18 0.71 0.00 820 0

 

 

Does anything look wierd>

 

Thanks for your help

[Geoff]

That looks ok to me, you really need to run iostat over an interval. Ideally when the server is running 'slow'. Also you can check the syslog to see if anything obvious was going on at the time it last run slow?

[tubaplayer83]

In the FOG Log Viewer under Multicast and Scheduler the logs are filled with "Failed to connect to database server, will try again in next iteration."

 

I'm trying to run a task now and the task is slowing down, it started at 4.5GiB/min and after a half an hour it's down to 1.24GiB/min

 

root@FOGSERVER:~# iostat

Linux 3.2.0-32-generic (FOGSERVER) 10/17/2012 _x86_64_ (2 CPU)

 

 

avg-cpu: %user %nice %system %iowait %steal %idle

0.62 0.14 1.28 4.40 0.00 93.56

 

 

Device: tps kB_read/s kB_wrtn/s kB_read kB_wrtn

sda 22.72 5290.91 32.64 15823256 97616

sdb 0.06 0.26 0.00 764 0

md0 0.06 0.25 0.00 748 0

dm-0 32.88 5290.00 32.64 15820549 97600

dm-1 0.07 0.27 0.00 820 12

 

I appreciate your help

[Geoff]

Ok I think your multicast imaging is broken. The error message indicates that fog can't access it's database. Are the credentials in the config file ok (look in /opt/fog/service/etc/config.php)?

[morganw]

I don't think this is a multicast issue if it is also causing speed issues on upload and computer wiping tasks as well. I would be interested to as to how the computer wipe has gone slower, there isn't a large data transfer involved here, it's just running shred locally. Is there a separate storage node, or just this single server?

[tubaplayer83]

Just the single server

[tubaplayer83]

Geoff, I checked the config.php file and I believe the credentials are ok. Could the credentials have changed randomly? I know for a fact that nobody has changed them on our end.

 

Thanks You for your help

[morganw]

Did anything else change? Did you run package upgrades, change the kernel or the init image?

Are your deployments multicast or unicast?

[tubaplayer83]

We have not changed a thing, our deployments are unicast

[morganw]

How about testing this all through a single switch, or VM to VM if the server is virtual? Have you experienced any slow down on anything else? e.g. DHCP lease request / allocation

[tubaplayer83]

Morgan,

 

We have tried running an image through a single switch and got the same result. The image speed continuously drops. I'm starting to think that the issue is with some sort of Ubunto configuration, the Linux expert from my office and I are having no luck figuring out what the issue is. We did notice in the system monitoring that the send speed is practically nothing, but we're not sure how to fix it. Do you have any ideas, I'm considering just rebuilding the server and using something other than Ubunto.

 

Thank You again for your assistance!

[morganw]

If you think it's the server then you could try mounting the NFS images share from something, then download an image file and see how long it takes. Then try the same thing from a FOG client in debug mode. If the first is fast and the second is slow then that would suggest the problem is the kernel on the client. If they are both slow then it's more likely a network problem (so you could retry through a basic unmanaged switch, after setting up a separate DHCP server to get things going), or something server side (look at packages that have been updated that could affect NIC performance). What type of network card is in the server? Also are you using Ubuntu with a desktop environment or just the text console?

[tubaplayer83]

Hi Morgan,

 

Sorry for taking so long to respond, things have been crazy over here. We setup a samba share on the fog server and placed a 4GB file on there. When I downloaded the file to my laptop it took less than a minute. Does that mean the problem is the kernel?

 

Thanks!

[morganw]

It implies that there isn't anything bizarre going on with the server, but you should probably re-test using NFS instead of Samba to match what happens during deployment. You can mount an NFS export with something like

mount -o nolock fogserver:/images /mnt/images

 

If your laptop runs Windows then you could boot it from a Linux live CD to do the test. Assuming you don't have any speed issues, you can then test the same thing using the FOG kernel and init image by choosing debug from the FOG boot menu and doing a similar mount command. Which version of FOG are you using and are you using the default kernel and default init image? If using FOG 0.32 a lot of people reverted to the init image from the previous release due to performance problems.

[tubaplayer83]

Hi Morgan,

 

So here's the latest, after changing the kernel when I run tasks on an HP8460 they run fine, the speed drops a bit but picks back up. However when I run tasks on an HP Probook 4420s the task starts at half of the speed and the speed continuously drops throughout the task. Ideas?

[tubaplayer83]

Hi Morgan, While doing some digging I found out that Ubuntu 12.04 isn't very compatible with FOG, could that be the issue?

[morganw]

Hi Morgan,

 

So here's the latest, after changing the kernel when I run tasks on an HP8460 they run fine, the speed drops a bit but picks back up. However when I run tasks on an HP Probook 4420s the task starts at half of the speed and the speed continuously drops throughout the task. Ideas?

 

Were you running the default kernel and then changing to an alternate one? Which version of FOG are you using? The fact you've changed the kernel and got a speed increase suggest that the problem is client side, but referring to your original post, why would the client side have suddenly changed? Did someone change something?

 

Hi Morgan, While doing some digging I found out that Ubuntu 12.04 isn't very compatible with FOG, could that be the issue?

 

I would think the imaging process wouldn't be affected, but the scheduler and multicast services might not be running because FOG init.d scripts for these services wouldn't be valid on Ubuntu 12.04 (I know they don't run on Debian Squeeze without fiddling). I don't think this would affect deployment speed if you could successfully deploy to begin with.

[tubaplayer83]

Hi Morgan,

 

If I change the init.gz will that effect the images I already have? Also are these instructions correct?

 

Thank you very much for your help

 

 

mv /tftpboot/fog/images/init.gz /tftpboot/fog/images/init.gz.32cd /optwget FOG - A Free Cloning Solution - Browse Files at SourceForge.net xzvf fog_0.29.tar.gz /opt/fog_0.29cp fog_0.29/packages/tftp/fog/images/init.gz /tftpboot/fog/images/

[morganw]

Yes, if you change the init image it will be used on every client (unlike the kernel which can be specified per computer). Most people only experience a positive change by reverting back to the version 0.29 init image though and the actual hardware support comes from the kernel. The formatting looks a little strange on what you've posted, but it's the right idea. Rename or move the old file, then put the equivalent file from FOG 0.29 in it's place. You probably want to match the file permissions and ownership as well.

[tubaplayer83]

Hi Morgan,

 

I think I've figured out the issue, when I upload an image of these machines, the image size is 250GB. I have run disk cleanup and deleted the other partitions however the image size is still very large. Is there a step that I'm missing?

 

Thanks!

[morganw]

Is it 250GB on the server? The images should be compressed. Check the file sizes on the server:

ls -lh /images

 

Also, what type of image is it (in FOG terms)?

Managing FOG - FOGProject Wiki

[wlainey]

I currently am having the same issue with an HP 4545s. The image size is about 250GB for some reason there is no compression. After realizing this i made my partition 22GB and set sysprep to expand the partition after imaging. Even with a 22GB image when i upload it uploads a 250GB image. This is with the partition set to single disk multi partition. If i set it to single disk single partition, it still makes a 250GB partition and the partition format is raw... I am going to try setting up a 10.04 with fog v0.29 to see if it is an issue with fog. I have previously done other machines on 0.29 successfully but my company switched to the 4545s at the same time we updated our fog server. The old machines are offsite so I have no successful machines to compare on the new version of fog.. I'll repost later today to let you know.

 

Is it 250GB on the server? The images should be compressed. Check the file sizes on the server:

ls -lh /images

 

Also, what type of image is it (in FOG terms)?

Managing FOG - FOGProject Wiki