localzuk Posted June 24, 2019 Posted June 24, 2019 Seems there is something going on with the internet today - large swathes of services are down or slow etc... Cloudflare are reporting a "routing leak". Anyone else seeing this? https://downdetector.co.uk/ is showing a nice grid of services experiencing issues.
bushby Posted June 24, 2019 Posted June 24, 2019 cloudflare have big issues atm https://www.theverge.com/2019/6/24/18715308/discord-down-outage-cloudflare-problems-crunchyroll-feedly?utm_campaign=theverge&utm_content=chorus&utm_medium=social&utm_source=twitter
6Foot2 Posted June 24, 2019 Posted June 24, 2019 dell.co.uk has been giving us issues today (back again now)
hardtailstar Posted June 24, 2019 Posted June 24, 2019 Quick someone contact the Internet Elders!!.... Not had any issues reported so far.
Garacesh Posted June 24, 2019 Posted June 24, 2019 (edited) Cloudflare are reporting a "routing leak". Sounds like traffic is getting "accidentally" routed through China. Again.. Edited June 24, 2019 by Garacesh
localzuk Posted June 24, 2019 Author Posted June 24, 2019 Sounds like traffic is getting "accidentally" routed through China again..That was my first thought lol
sparkeh Posted June 24, 2019 Posted June 24, 2019 Yeah general all round weirdness today. Also... Discord!?
Garacesh Posted June 24, 2019 Posted June 24, 2019 (edited) https://www.theregister.co.uk/2019/06/24/cloudflare_route_leak/ Not really much info, but the article might expand as time goes on. Yeah general all round weirdness today. Also... Discord!? Lol yep that was how I noticed it. Discord asplode. Edit: Cloudflare have declared the issue 'resolved' Route Leak Impacting Cloudflare Resolved - Traffic levels have returned to normal now that the route leak has been fixed. We are now marking this incident as resolved. Jun 24, 13:02 UTC [Source - Cloudflare Status page] Except services are still down. Edited June 24, 2019 by Garacesh
jthompson Posted June 24, 2019 Posted June 24, 2019 Feedly wasn't available when I was 'at lunch'. Had to resort to browsing around on actual websites.
howartp Posted June 24, 2019 Posted June 24, 2019 Edugeek itself went skewiff around 12.45 for us. Other sites were ok.
pete Posted June 25, 2019 Posted June 25, 2019 Cloudflare post-incident write-up. TLDR: Verizon are incompetent muppets. https://blog.cloudflare.com/how-verizon-and-a-bgp-optimizer-knocked-large-parts-of-the-internet-offline-today/ 1
Garacesh Posted July 2, 2019 Posted July 2, 2019 I'm noticing a trend again, that it's the Discord users spotting this 1
Arthur Posted July 3, 2019 Posted July 3, 2019 Yesterday's 30 minute Cloudflare outage was caused by a bad regular expression. 🤔 Cloudflare outage caused by bad software deploy Starting at 13:42 UTC today we experienced a global outage across our network that resulted in visitors to Cloudflare-proxied domains being shown 502 errors ("Bad Gateway"). The cause of this outage was deployment of a single misconfigured rule within the Cloudflare Web Application Firewall (WAF) during a routine deployment of new Cloudflare WAF Managed rules. The intent of these new rules was to improve the blocking of inline JavaScript that is used in attacks. These rules were being deployed in a simulated mode where issues are identified and logged by the new rule but no customer traffic is actually blocked so that we can measure false positive rates and ensure that the new rules do not cause problems when they are deployed into full production. Unfortunately, one of these rules contained a regular expression that caused CPU to spike to 100% on our machines worldwide. This 100% CPU spike caused the 502 errors that our customers saw. At its worst traffic dropped by 82%. 1
XiJ Posted July 3, 2019 Posted July 3, 2019 Yesterday's 30 minute Cloudflare outage was caused by a bad regular expression. [emoji848] “Oh sh**” ?!
bald_pig Posted July 3, 2019 Posted July 3, 2019 That's nothing, 83 DECILLION IP address were wrongly announced for a week and nobody noticed... because someone missed a 7. This major internet routing blunder took A WEEK to fix. Why so long? It was IPv6 – and no one really noticed 2
sparkeh Posted July 3, 2019 Posted July 3, 2019 That's nothing, 83 DECILLION IP address were wrongly announced for a week and nobody noticed... because someone missed a 7. This major internet routing blunder took A WEEK to fix. Why so long? It was IPv6 – and no one really noticed Shows how well IPv6 adoption is going...
Arthur Posted July 12, 2019 Posted July 12, 2019 Yesterday's 30 minute Cloudflare outage was caused by a bad regular expression. Cloudflare have posted a full post-mortem on the recent outage.... https://blog.cloudflare.com/details-of-the-cloudflare-outage-on-july-2-2019 To fully understand how [b](??:"|'|\]|\}|\\|\d|(?:nan|infinity|true|false|null|undefined|symbol|math)|\`|\-|\+)+[)]*;?((?:\s|-|~|!|{}|\|\||\+)*.*(?:.*=.*)))[/b] caused CPU exhaustion you need to understand a little about how a standard regular expression engine works. The critical part is .*(?:.*=.*). The (?: and matching ) are a non-capturing group (i.e. the expression inside the parentheses is grouped together as a single expression). For the purposes of the discussion of why this pattern causes CPU exhaustion we can safely ignore it and treat the pattern as .*.*=.*. When reduced to this, the pattern obviously looks unnecessarily complex; but what's important is any "real-world" expression (like the complex ones in our WAF rules) that ask the engine to "match anything followed by anything" can lead to catastrophic backtracking. Here’s why. The blog post is an interesting read if you use regular expressions. 1
Recommended Posts
Create an account or sign in to comment
You need to be a member in order to leave a comment
Create an account
Sign up for a new account in our community. It's easy!
Register a new accountSign in
Already have an account? Sign in here.
Sign In Now