tubaplayer83 Posted October 17, 2012 Posted October 17, 2012 (edited) Hello, We have been using our FOG server for a few months now and it has been running great. Since OCT 5th all tasks that have been run have taken over 7 hours to complete, this includes image uploading, deploying images and wiping a computer. The tasks start out running over 4GB/min and gradually slow down to under 500mb/min Does anyone have any idea what could be causing this? Thank you for your help Edited October 17, 2012 by ZeroHour
Geoff Posted October 17, 2012 Posted October 17, 2012 Use ifconfig to check if there are any errors reported on the network interface. Use iostat to see if there are any I/O load issues 1
tubaplayer83 Posted October 17, 2012 Author Posted October 17, 2012 I ran ifconfig and there were no errors reported, when i ran iostat it came up with bin/sh: iostat: not found. I aplogize, I'm kind of learning all of this on the fly, did I do that wrong?
tubaplayer83 Posted October 17, 2012 Author Posted October 17, 2012 OK, I figured it out. ifconfig UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:5470 errors:0 dropped:0 overruns:0 frame:0 TX packets:526 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:1000414 (1.0 MB) TX bytes:102863 (102.8 KB) Interrupt:21 Memory:dffe0000-e0000000 lo Link encap:Local Loopback inet addr:127.0.0.1 Mask:255.0.0.0 inet6 addr: ::1/128 Scope:Host UP LOOPBACK RUNNING MTU:16436 Metric:1 RX packets:68 errors:0 dropped:0 overruns:0 frame:0 TX packets:68 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:5070 (5.0 KB) TX bytes:5070 (5.0 KB) virbr0 Link encap:Ethernet HWaddr d2:cf:68:16:33:80 inet addr:192.168.122.1 Bcast:192.168.122.255 Mask:255.255.255.0 UP BROADCAST MULTICAST MTU:1500 Metric:1 RX packets:0 errors:0 dropped:0 overruns:0 frame:0 TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:0 (0.0 B) TX bytes:0 (0.0 B) iostat root@FOGSERVER:~# iostat Linux 3.2.0-32-generic (FOGSERVER) 10/17/2012 _x86_64_ (2 CPU) avg-cpu: %user %nice %system %iowait %steal %idle 1.35 0.36 0.81 1.64 0.00 95.84 Device: tps kB_read/s kB_wrtn/s kB_read kB_wrtn sda 13.81 473.00 76.95 550108 89492 sdb 0.16 0.66 0.00 764 0 md0 0.16 0.64 0.00 748 0 dm-0 24.26 470.66 76.94 547393 89488 dm-1 0.18 0.71 0.00 820 0 Does anything look wierd> Thanks for your help
Geoff Posted October 17, 2012 Posted October 17, 2012 That looks ok to me, you really need to run iostat over an interval. Ideally when the server is running 'slow'. Also you can check the syslog to see if anything obvious was going on at the time it last run slow? 1
tubaplayer83 Posted October 17, 2012 Author Posted October 17, 2012 In the FOG Log Viewer under Multicast and Scheduler the logs are filled with "Failed to connect to database server, will try again in next iteration." I'm trying to run a task now and the task is slowing down, it started at 4.5GiB/min and after a half an hour it's down to 1.24GiB/min root@FOGSERVER:~# iostat Linux 3.2.0-32-generic (FOGSERVER) 10/17/2012 _x86_64_ (2 CPU) avg-cpu: %user %nice %system %iowait %steal %idle 0.62 0.14 1.28 4.40 0.00 93.56 Device: tps kB_read/s kB_wrtn/s kB_read kB_wrtn sda 22.72 5290.91 32.64 15823256 97616 sdb 0.06 0.26 0.00 764 0 md0 0.06 0.25 0.00 748 0 dm-0 32.88 5290.00 32.64 15820549 97600 dm-1 0.07 0.27 0.00 820 12 I appreciate your help
Geoff Posted October 17, 2012 Posted October 17, 2012 Ok I think your multicast imaging is broken. The error message indicates that fog can't access it's database. Are the credentials in the config file ok (look in /opt/fog/service/etc/config.php)? 1
morganw Posted October 17, 2012 Posted October 17, 2012 I don't think this is a multicast issue if it is also causing speed issues on upload and computer wiping tasks as well. I would be interested to as to how the computer wipe has gone slower, there isn't a large data transfer involved here, it's just running shred locally. Is there a separate storage node, or just this single server? 1
tubaplayer83 Posted October 18, 2012 Author Posted October 18, 2012 Geoff, I checked the config.php file and I believe the credentials are ok. Could the credentials have changed randomly? I know for a fact that nobody has changed them on our end. Thanks You for your help
morganw Posted October 18, 2012 Posted October 18, 2012 Did anything else change? Did you run package upgrades, change the kernel or the init image? Are your deployments multicast or unicast? 1
tubaplayer83 Posted October 18, 2012 Author Posted October 18, 2012 We have not changed a thing, our deployments are unicast
morganw Posted October 18, 2012 Posted October 18, 2012 How about testing this all through a single switch, or VM to VM if the server is virtual? Have you experienced any slow down on anything else? e.g. DHCP lease request / allocation 1
tubaplayer83 Posted October 19, 2012 Author Posted October 19, 2012 Morgan, We have tried running an image through a single switch and got the same result. The image speed continuously drops. I'm starting to think that the issue is with some sort of Ubunto configuration, the Linux expert from my office and I are having no luck figuring out what the issue is. We did notice in the system monitoring that the send speed is practically nothing, but we're not sure how to fix it. Do you have any ideas, I'm considering just rebuilding the server and using something other than Ubunto. Thank You again for your assistance!
morganw Posted October 19, 2012 Posted October 19, 2012 If you think it's the server then you could try mounting the NFS images share from something, then download an image file and see how long it takes. Then try the same thing from a FOG client in debug mode. If the first is fast and the second is slow then that would suggest the problem is the kernel on the client. If they are both slow then it's more likely a network problem (so you could retry through a basic unmanaged switch, after setting up a separate DHCP server to get things going), or something server side (look at packages that have been updated that could affect NIC performance). What type of network card is in the server? Also are you using Ubuntu with a desktop environment or just the text console?
tubaplayer83 Posted October 25, 2012 Author Posted October 25, 2012 Hi Morgan, Sorry for taking so long to respond, things have been crazy over here. We setup a samba share on the fog server and placed a 4GB file on there. When I downloaded the file to my laptop it took less than a minute. Does that mean the problem is the kernel? Thanks!
morganw Posted October 26, 2012 Posted October 26, 2012 It implies that there isn't anything bizarre going on with the server, but you should probably re-test using NFS instead of Samba to match what happens during deployment. You can mount an NFS export with something like mount -o nolock fogserver:/images /mnt/images If your laptop runs Windows then you could boot it from a Linux live CD to do the test. Assuming you don't have any speed issues, you can then test the same thing using the FOG kernel and init image by choosing debug from the FOG boot menu and doing a similar mount command. Which version of FOG are you using and are you using the default kernel and default init image? If using FOG 0.32 a lot of people reverted to the init image from the previous release due to performance problems.
tubaplayer83 Posted October 30, 2012 Author Posted October 30, 2012 Hi Morgan, So here's the latest, after changing the kernel when I run tasks on an HP8460 they run fine, the speed drops a bit but picks back up. However when I run tasks on an HP Probook 4420s the task starts at half of the speed and the speed continuously drops throughout the task. Ideas?
tubaplayer83 Posted October 30, 2012 Author Posted October 30, 2012 Hi Morgan, While doing some digging I found out that Ubuntu 12.04 isn't very compatible with FOG, could that be the issue?
morganw Posted October 30, 2012 Posted October 30, 2012 Hi Morgan, So here's the latest, after changing the kernel when I run tasks on an HP8460 they run fine, the speed drops a bit but picks back up. However when I run tasks on an HP Probook 4420s the task starts at half of the speed and the speed continuously drops throughout the task. Ideas? Were you running the default kernel and then changing to an alternate one? Which version of FOG are you using? The fact you've changed the kernel and got a speed increase suggest that the problem is client side, but referring to your original post, why would the client side have suddenly changed? Did someone change something? Hi Morgan, While doing some digging I found out that Ubuntu 12.04 isn't very compatible with FOG, could that be the issue? I would think the imaging process wouldn't be affected, but the scheduler and multicast services might not be running because FOG init.d scripts for these services wouldn't be valid on Ubuntu 12.04 (I know they don't run on Debian Squeeze without fiddling). I don't think this would affect deployment speed if you could successfully deploy to begin with.
tubaplayer83 Posted November 1, 2012 Author Posted November 1, 2012 Hi Morgan, If I change the init.gz will that effect the images I already have? Also are these instructions correct? Thank you very much for your help mv /tftpboot/fog/images/init.gz /tftpboot/fog/images/init.gz.32cd /optwget FOG - A Free Cloning Solution - Browse Files at SourceForge.net xzvf fog_0.29.tar.gz /opt/fog_0.29cp fog_0.29/packages/tftp/fog/images/init.gz /tftpboot/fog/images/
morganw Posted November 1, 2012 Posted November 1, 2012 Yes, if you change the init image it will be used on every client (unlike the kernel which can be specified per computer). Most people only experience a positive change by reverting back to the version 0.29 init image though and the actual hardware support comes from the kernel. The formatting looks a little strange on what you've posted, but it's the right idea. Rename or move the old file, then put the equivalent file from FOG 0.29 in it's place. You probably want to match the file permissions and ownership as well.
tubaplayer83 Posted November 9, 2012 Author Posted November 9, 2012 Hi Morgan, I think I've figured out the issue, when I upload an image of these machines, the image size is 250GB. I have run disk cleanup and deleted the other partitions however the image size is still very large. Is there a step that I'm missing? Thanks!
morganw Posted November 9, 2012 Posted November 9, 2012 Is it 250GB on the server? The images should be compressed. Check the file sizes on the server: ls -lh /images Also, what type of image is it (in FOG terms)? Managing FOG - FOGProject Wiki
wlainey Posted November 12, 2012 Posted November 12, 2012 I currently am having the same issue with an HP 4545s. The image size is about 250GB for some reason there is no compression. After realizing this i made my partition 22GB and set sysprep to expand the partition after imaging. Even with a 22GB image when i upload it uploads a 250GB image. This is with the partition set to single disk multi partition. If i set it to single disk single partition, it still makes a 250GB partition and the partition format is raw... I am going to try setting up a 10.04 with fog v0.29 to see if it is an issue with fog. I have previously done other machines on 0.29 successfully but my company switched to the 4545s at the same time we updated our fog server. The old machines are offsite so I have no successful machines to compare on the new version of fog.. I'll repost later today to let you know. Is it 250GB on the server? The images should be compressed. Check the file sizes on the server: ls -lh /images Also, what type of image is it (in FOG terms)? Managing FOG - FOGProject Wiki
Recommended Posts
Create an account or sign in to comment
You need to be a member in order to leave a comment
Create an account
Sign up for a new account in our community. It's easy!
Register a new accountSign in
Already have an account? Sign in here.
Sign In Now