[Fog] Tasks Slowing Down

[tubaplayer83]

Hello,

We have been using our FOG server for a few months now and it has been running great. Since OCT 5th all tasks that have been run have taken over 7 hours to complete, this includes image uploading, deploying images and wiping a computer. The tasks start out running over 4GB/min and gradually slow down to under 500mb/min

Does anyone have any idea what could be causing this?

Thank you for your help

[Geoff]

Use ifconfig to check if there are any errors reported on the network interface. Use iostat to see if there are any I/O load issues

[tubaplayer83]

I ran ifconfig and there were no errors reported, when i ran iostat it came up with bin/sh: iostat: not found. I aplogize, I'm kind of learning all of this on the fly, did I do that wrong?

[tubaplayer83]

OK, I figured it out.

ifconfig

UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:5470 errors:0 dropped:0 overruns:0 frame:0
TX packets:526 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:1000414 (1.0 MB) TX bytes:102863 (102.8 KB)
Interrupt:21 Memory:dffe0000-e0000000


lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:16436 Metric:1
RX packets:68 errors:0 dropped:0 overruns:0 frame:0
TX packets:68 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:5070 (5.0 KB) TX bytes:5070 (5.0 KB)


virbr0 Link encap:Ethernet HWaddr d2:cf:68:16:33:80
inet addr:192.168.122.1 Bcast:192.168.122.255 Mask:255.255.255.0
UP BROADCAST MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)

iostat

root@FOGSERVER:~# iostat
Linux 3.2.0-32-generic (FOGSERVER) 10/17/2012 _x86_64_ (2 CPU)


avg-cpu: %user %nice %system %iowait %steal %idle
1.35 0.36 0.81 1.64 0.00 95.84


Device: tps kB_read/s kB_wrtn/s kB_read kB_wrtn
sda 13.81 473.00 76.95 550108 89492
sdb 0.16 0.66 0.00 764 0
md0 0.16 0.64 0.00 748 0
dm-0 24.26 470.66 76.94 547393 89488
dm-1 0.18 0.71 0.00 820 0


Does anything look wierd>

Thanks for your help

[Geoff]

That looks ok to me, you really need to run iostat over an interval. Ideally when the server is running 'slow'. Also you can check the syslog to see if anything obvious was going on at the time it last run slow?

[tubaplayer83]

In the FOG Log Viewer under Multicast and Scheduler the logs are filled with "Failed to connect to database server, will try again in next iteration."

I'm trying to run a task now and the task is slowing down, it started at 4.5GiB/min and after a half an hour it's down to 1.24GiB/min

root@FOGSERVER:~# iostat
Linux 3.2.0-32-generic (FOGSERVER) 10/17/2012 _x86_64_ (2 CPU)


avg-cpu: %user %nice %system %iowait %steal %idle
0.62 0.14 1.28 4.40 0.00 93.56


Device: tps kB_read/s kB_wrtn/s kB_read kB_wrtn
sda 22.72 5290.91 32.64 15823256 97616
sdb 0.06 0.26 0.00 764 0
md0 0.06 0.25 0.00 748 0
dm-0 32.88 5290.00 32.64 15820549 97600
dm-1 0.07 0.27 0.00 820 12

I appreciate your help

[Geoff]

Ok I think your multicast imaging is broken. The error message indicates that fog can't access it's database. Are the credentials in the config file ok (look in /opt/fog/service/etc/config.php)?

[morganw]

I don't think this is a multicast issue if it is also causing speed issues on upload and computer wiping tasks as well. I would be interested to as to how the computer wipe has gone slower, there isn't a large data transfer involved here, it's just running shred locally. Is there a separate storage node, or just this single server?

[tubaplayer83]

Just the single server

[tubaplayer83]

Geoff, I checked the config.php file and I believe the credentials are ok. Could the credentials have changed randomly? I know for a fact that nobody has changed them on our end.

Thanks You for your help

[morganw]

Did anything else change? Did you run package upgrades, change the kernel or the init image?
Are your deployments multicast or unicast?

[tubaplayer83]

We have not changed a thing, our deployments are unicast

[morganw]

How about testing this all through a single switch, or VM to VM if the server is virtual? Have you experienced any slow down on anything else? e.g. DHCP lease request / allocation

[tubaplayer83]

Morgan,

We have tried running an image through a single switch and got the same result. The image speed continuously drops. I'm starting to think that the issue is with some sort of Ubunto configuration, the Linux expert from my office and I are having no luck figuring out what the issue is. We did notice in the system monitoring that the send speed is practically nothing, but we're not sure how to fix it. Do you have any ideas, I'm considering just rebuilding the server and using something other than Ubunto.

Thank You again for your assistance!

[morganw]

If you think it's the server then you could try mounting the NFS images share from something, then download an image file and see how long it takes. Then try the same thing from a FOG client in debug mode. If the first is fast and the second is slow then that would suggest the problem is the kernel on the client. If they are both slow then it's more likely a network problem (so you could retry through a basic unmanaged switch, after setting up a separate DHCP server to get things going), or something server side (look at packages that have been updated that could affect NIC performance). What type of network card is in the server? Also are you using Ubuntu with a desktop environment or just the text console?