+ Post New Thread
Page 1 of 2 12 LastLast
Results 1 to 15 of 27
O/S Deployment Thread, Tasks Slowing Down in Technical; Hello, We have been using our FOG server for a few months now and it has been running great. Since ...
  1. #1

    Join Date
    Oct 2012
    Posts
    13
    Thank Post
    6
    Thanked 0 Times in 0 Posts
    Rep Power
    0

    Tasks Slowing Down

    Hello,

    We have been using our FOG server for a few months now and it has been running great. Since OCT 5th all tasks that have been run have taken over 7 hours to complete, this includes image uploading, deploying images and wiping a computer. The tasks start out running over 4GB/min and gradually slow down to under 500mb/min

    Does anyone have any idea what could be causing this?

    Thank you for your help
    Last edited by ZeroHour; 17th October 2012 at 05:55 PM.

  2. #2

    Geoff's Avatar
    Join Date
    Jun 2005
    Location
    Fylde, Lancs, UK.
    Posts
    11,813
    Thank Post
    110
    Thanked 586 Times in 507 Posts
    Blog Entries
    1
    Rep Power
    225
    Use ifconfig to check if there are any errors reported on the network interface. Use iostat to see if there are any I/O load issues

  3. Thanks to Geoff from:

    tubaplayer83 (19th October 2012)

  4. #3

    Join Date
    Oct 2012
    Posts
    13
    Thank Post
    6
    Thanked 0 Times in 0 Posts
    Rep Power
    0
    I ran ifconfig and there were no errors reported, when i ran iostat it came up with bin/sh: iostat: not found. I aplogize, I'm kind of learning all of this on the fly, did I do that wrong?

  5. #4

    Join Date
    Oct 2012
    Posts
    13
    Thank Post
    6
    Thanked 0 Times in 0 Posts
    Rep Power
    0
    OK, I figured it out.

    ifconfig

    UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
    RX packets:5470 errors:0 dropped:0 overruns:0 frame:0
    TX packets:526 errors:0 dropped:0 overruns:0 carrier:0
    collisions:0 txqueuelen:1000
    RX bytes:1000414 (1.0 MB) TX bytes:102863 (102.8 KB)
    Interrupt:21 Memory:dffe0000-e0000000


    lo Link encap:Local Loopback
    inet addr:127.0.0.1 Mask:255.0.0.0
    inet6 addr: ::1/128 Scope:Host
    UP LOOPBACK RUNNING MTU:16436 Metric:1
    RX packets:68 errors:0 dropped:0 overruns:0 frame:0
    TX packets:68 errors:0 dropped:0 overruns:0 carrier:0
    collisions:0 txqueuelen:0
    RX bytes:5070 (5.0 KB) TX bytes:5070 (5.0 KB)


    virbr0 Link encap:Ethernet HWaddr d2:cf:68:16:33:80
    inet addr:192.168.122.1 Bcast:192.168.122.255 Mask:255.255.255.0
    UP BROADCAST MULTICAST MTU:1500 Metric:1
    RX packets:0 errors:0 dropped:0 overruns:0 frame:0
    TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
    collisions:0 txqueuelen:0
    RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)

    iostat

    root@FOGSERVER:~# iostat
    Linux 3.2.0-32-generic (FOGSERVER) 10/17/2012 _x86_64_ (2 CPU)


    avg-cpu: %user %nice %system %iowait %steal %idle
    1.35 0.36 0.81 1.64 0.00 95.84


    Device: tps kB_read/s kB_wrtn/s kB_read kB_wrtn
    sda 13.81 473.00 76.95 550108 89492
    sdb 0.16 0.66 0.00 764 0
    md0 0.16 0.64 0.00 748 0
    dm-0 24.26 470.66 76.94 547393 89488
    dm-1 0.18 0.71 0.00 820 0


    Does anything look wierd>

    Thanks for your help

  6. #5

    Geoff's Avatar
    Join Date
    Jun 2005
    Location
    Fylde, Lancs, UK.
    Posts
    11,813
    Thank Post
    110
    Thanked 586 Times in 507 Posts
    Blog Entries
    1
    Rep Power
    225
    That looks ok to me, you really need to run iostat over an interval. Ideally when the server is running 'slow'. Also you can check the syslog to see if anything obvious was going on at the time it last run slow?

  7. Thanks to Geoff from:

    tubaplayer83 (19th October 2012)

  8. #6

    Join Date
    Oct 2012
    Posts
    13
    Thank Post
    6
    Thanked 0 Times in 0 Posts
    Rep Power
    0
    In the FOG Log Viewer under Multicast and Scheduler the logs are filled with "Failed to connect to database server, will try again in next iteration."

    I'm trying to run a task now and the task is slowing down, it started at 4.5GiB/min and after a half an hour it's down to 1.24GiB/min

    root@FOGSERVER:~# iostat
    Linux 3.2.0-32-generic (FOGSERVER) 10/17/2012 _x86_64_ (2 CPU)


    avg-cpu: %user %nice %system %iowait %steal %idle
    0.62 0.14 1.28 4.40 0.00 93.56


    Device: tps kB_read/s kB_wrtn/s kB_read kB_wrtn
    sda 22.72 5290.91 32.64 15823256 97616
    sdb 0.06 0.26 0.00 764 0
    md0 0.06 0.25 0.00 748 0
    dm-0 32.88 5290.00 32.64 15820549 97600
    dm-1 0.07 0.27 0.00 820 12

    I appreciate your help

  9. #7

    Geoff's Avatar
    Join Date
    Jun 2005
    Location
    Fylde, Lancs, UK.
    Posts
    11,813
    Thank Post
    110
    Thanked 586 Times in 507 Posts
    Blog Entries
    1
    Rep Power
    225
    Ok I think your multicast imaging is broken. The error message indicates that fog can't access it's database. Are the credentials in the config file ok (look in /opt/fog/service/etc/config.php)?

  10. Thanks to Geoff from:

    tubaplayer83 (19th October 2012)

  11. #8
    morganw's Avatar
    Join Date
    Apr 2009
    Location
    Cambridge
    Posts
    816
    Thank Post
    46
    Thanked 132 Times in 126 Posts
    Rep Power
    40
    I don't think this is a multicast issue if it is also causing speed issues on upload and computer wiping tasks as well. I would be interested to as to how the computer wipe has gone slower, there isn't a large data transfer involved here, it's just running shred locally. Is there a separate storage node, or just this single server?

  12. Thanks to morganw from:

    tubaplayer83 (19th October 2012)

  13. #9

    Join Date
    Oct 2012
    Posts
    13
    Thank Post
    6
    Thanked 0 Times in 0 Posts
    Rep Power
    0
    Just the single server

  14. #10

    Join Date
    Oct 2012
    Posts
    13
    Thank Post
    6
    Thanked 0 Times in 0 Posts
    Rep Power
    0
    Geoff, I checked the config.php file and I believe the credentials are ok. Could the credentials have changed randomly? I know for a fact that nobody has changed them on our end.

    Thanks You for your help

  15. #11
    morganw's Avatar
    Join Date
    Apr 2009
    Location
    Cambridge
    Posts
    816
    Thank Post
    46
    Thanked 132 Times in 126 Posts
    Rep Power
    40
    Did anything else change? Did you run package upgrades, change the kernel or the init image?
    Are your deployments multicast or unicast?

  16. Thanks to morganw from:

    tubaplayer83 (19th October 2012)

  17. #12

    Join Date
    Oct 2012
    Posts
    13
    Thank Post
    6
    Thanked 0 Times in 0 Posts
    Rep Power
    0
    We have not changed a thing, our deployments are unicast

  18. #13
    morganw's Avatar
    Join Date
    Apr 2009
    Location
    Cambridge
    Posts
    816
    Thank Post
    46
    Thanked 132 Times in 126 Posts
    Rep Power
    40
    How about testing this all through a single switch, or VM to VM if the server is virtual? Have you experienced any slow down on anything else? e.g. DHCP lease request / allocation

  19. Thanks to morganw from:

    tubaplayer83 (19th October 2012)

  20. #14

    Join Date
    Oct 2012
    Posts
    13
    Thank Post
    6
    Thanked 0 Times in 0 Posts
    Rep Power
    0
    Morgan,

    We have tried running an image through a single switch and got the same result. The image speed continuously drops. I'm starting to think that the issue is with some sort of Ubunto configuration, the Linux expert from my office and I are having no luck figuring out what the issue is. We did notice in the system monitoring that the send speed is practically nothing, but we're not sure how to fix it. Do you have any ideas, I'm considering just rebuilding the server and using something other than Ubunto.

    Thank You again for your assistance!

  21. #15
    morganw's Avatar
    Join Date
    Apr 2009
    Location
    Cambridge
    Posts
    816
    Thank Post
    46
    Thanked 132 Times in 126 Posts
    Rep Power
    40
    If you think it's the server then you could try mounting the NFS images share from something, then download an image file and see how long it takes. Then try the same thing from a FOG client in debug mode. If the first is fast and the second is slow then that would suggest the problem is the kernel on the client. If they are both slow then it's more likely a network problem (so you could retry through a basic unmanaged switch, after setting up a separate DHCP server to get things going), or something server side (look at packages that have been updated that could affect NIC performance). What type of network card is in the server? Also are you using Ubuntu with a desktop environment or just the text console?

SHARE:
+ Post New Thread
Page 1 of 2 12 LastLast

Similar Threads

  1. Network Slow Down in One Classroom
    By FragglePete in forum Wireless Networks
    Replies: 10
    Last Post: 4th October 2008, 01:54 PM
  2. Slowing down....
    By jcollings in forum Wireless Networks
    Replies: 9
    Last Post: 20th August 2008, 02:53 PM
  3. [CLOSED] Bug/Error: Adverts slowing down loading?
    By localzuk in forum EduGeek.net Site Problems
    Replies: 13
    Last Post: 6th April 2008, 10:40 PM
  4. Site slow down - SE spiders
    By contink in forum Comments and Suggestions
    Replies: 3
    Last Post: 15th June 2007, 05:26 AM

Thread Information

Users Browsing this Thread

There are currently 1 users browsing this thread. (0 members and 1 guests)

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •