+ Post New Thread
Results 1 to 5 of 5
Internet Related/Filtering/Firewall Thread, Odd Internet goings on yesterday in Technical; So who saw some of the Internet drop off the face of the planet yesterday morning??? Many ISP's around the ...
  1. #1

    Join Date
    Apr 2012
    Location
    Leeds
    Posts
    343
    Thank Post
    3
    Thanked 72 Times in 57 Posts
    Rep Power
    38

    Exclamation Odd Internet goings on yesterday

    So who saw some of the Internet drop off the face of the planet yesterday morning???

    Many ISP's around the world were affected.

    Unfortunately we did as one of our smaller transit providers unfortunately hit the magic 524,288 BGP routes in its edge router BGP table and then started to delete certain new ones as the memory allocated for them filled up as they were using Cisco 7600 series routers (for many years the workhorse of many ISP BGP edge routers) which can't take more than 512k routes unless IPv6 is disabled on them :-/

    So they got turned off.

    A brief description of what happened is below from our Technical Director Matthew Hattersley with a link giving a full explanation further below.

    Internet Touches Half Million Routes: Outages Possible Next Week - Renesys


    Essentially the issue here is the amount of memory that routers allocate to storing routes. Certain models of routers and line cards have a hard limit of 512k (Some even lower, but they are mostly gone nowadays). Others have a soft limit.

    A good example is the Cisco 7600 (With a RSP720-3CXL). This router for many years has been the workhorse of many ISP BGP edges. It hardy and reliable but as its been so reliable, a lot of operators have taken it for granted. By default the RSP720-3CXL allocates enough memory for 512k IPv4 routes. Leaving the leftover for IPv6 and other rarer kinds of routes (Multicast etc). Yesterday this limit was hit for a short period. At which point these routers essentially ran out of memory to store routes and starting dropping them. This meant different things for different operators as the routes dropped was largely random (Imagine loosing the default route to propagate around your network. Essentially the loss of one route looses everything).

    As regards the impact to Talk Straight, directly there was no effect. Our core is based upon the Cisco ASR9000range of routers, capable of around 4 million routes. However like everyone else we saw issues getting to certain upstream peers. For us this meant our customers couldn't get to Office 365 and RackSpace. Two major online applications/networks that our client base rely on.

    This was traced to a 3rd party 7600 series router. In order to fix this issue the upstream provider (Like many others) reallocated their memory to give more space to IPv4 routes (sacrificing IPv6 in the process).

    What does this mean going forwards?

    Well for Talk Straight / Schools Broadband, not a lot. We'll probably remove that upstream peer from our routing mix to avoid issues in the future. For the internet at large the people running these older platforms have two options. Replace their equipment or sacrifice IPv6 routing. We'll probably see a mix over the next few months, but eventually they will have to remove this kit as IPv6 isn't going anywhere."



    If any of you have any questions don't hesitate to get in touch.

    Thanks

    Dave

  2. 3 Thanks to SchoolsBroadband:

    Dos_Box (13th August 2014), SimonD (13th August 2014), VeryPC_Colin_M (13th August 2014)

  3. #2

    DaveP's Avatar
    Join Date
    Oct 2006
    Location
    Can't talk now: The mother-ship is calling!
    Posts
    9,365
    Thank Post
    353
    Thanked 1,348 Times in 926 Posts
    Blog Entries
    4
    Rep Power
    1202
    A post on a similar theme:

    Link: Nobody will ever need more than 512K for routing....

  4. #3

    Dos_Box's Avatar
    Join Date
    Jun 2005
    Location
    Preston, Lancashire
    Posts
    9,435
    Thank Post
    701
    Thanked 2,302 Times in 1,063 Posts
    Blog Entries
    23
    Rep Power
    678
    This actually goes a long way to explaining why some quite large sites were down yesterday (lots of eBay EU sites for example).

  5. #4
    MrMat's Avatar
    Join Date
    Jul 2013
    Location
    Solihull
    Posts
    183
    Thank Post
    20
    Thanked 40 Times in 30 Posts
    Rep Power
    28
    Don't try and fool us! We know what happened really. Jen turned it off didn't she?!

    itcrowd_internet_box.jpg

  6. 2 Thanks to MrMat:

    Cache (13th August 2014), TheScarfedOne (13th August 2014)

  7. #5
    cpjitservices's Avatar
    Join Date
    Jul 2010
    Location
    Hessle
    Posts
    2,605
    Thank Post
    544
    Thanked 301 Times in 277 Posts
    Rep Power
    85
    Sounds like the issue from yesteryear when Juniper released an update that caused a core dump failure.... could be a similar thing again.



SHARE:
+ Post New Thread

Similar Threads

  1. Replies: 4
    Last Post: 6th September 2007, 10:22 AM
  2. Replies: 9
    Last Post: 12th August 2007, 02:35 PM
  3. Were all going on asummer holiday (aircon)
    By russdev in forum Wireless Networks
    Replies: 10
    Last Post: 18th June 2007, 03:56 PM
  4. TV Over Internet / Video-on-Demand
    By enjay in forum IT News
    Replies: 19
    Last Post: 11th May 2007, 03:04 PM

Thread Information

Users Browsing this Thread

There are currently 1 users browsing this thread. (0 members and 1 guests)

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •