AWS issues this morning.

[RLR]

Lots of online services affected by this it seems. Happy Monday.

[6Foot3]

https://www.bbc.co.uk/news/live/c5y8k7k6v1rt

[machy]

Relevant XKCD 

[NegativeKillDeath]

https://www.theguardian.com/business/live/2025/oct/20/china-economic-growth-gdp-low-house-prices-fall-stock-markets-ftse-business-live-news?CMP=share_btn_url&page=with%3Ablock-68f5fb228f089def4b0732d6#block-68f5fb228f089def4b0732d6

 

 

Always DNS.

 

[tom_newton]

Even if you arent on AWS (we're either GCP, or Azure here) you can't escape. Some of the 3rd parties we use have AWS under the hood - including folks like zendesk, and so we haven't had a fully normal day. Luckily super important stuff, like filtering, is designed for mega resilience to this sort of stuff.

 

I had my money on power, but so far DNS seems to be winning as the root cause (but what caused the dns issue?)

[dmj]

22 minutes ago, tom_newton said:

Even if you arent on AWS (we're either GCP, or Azure here) you can't escape. Some of the 3rd parties we use have AWS under the hood - including folks like zendesk, and so we haven't had a fully normal day. Luckily super important stuff, like filtering, is designed for mega resilience to this sort of stuff.

 

I had my money on power, but so far DNS seems to be winning as the root cause (but what caused the dns issue?)

100%.  loads of our third parties were down too.

 

Apparently it was DynamoDB that went down. IIRC last time this happened it took CloudFlare out, which did cause massive issues but it looks like CF fixed that bug.

[tom_newton]

But it was DNS which took out dynamo, I think

[6Foot2]

Link: AWS crash causes $2,000 Smart Beds to overheat and get stuck upright - Dexerto

 

Quote

A major Amazon Web Services (AWS) outage on October 20 had the unexpected side effect of causing chaos in bedrooms across the US, as owners of Eight Sleep’s $2,000+ ‘Pod’ mattress covers found their smart beds had no offline mode and were stuck at high temperatures and odd positions in the night.

 

The outage began around 3 am ET, when AWS reported “increased error rates and latencies” in its US-EAST-1 region. By mid-morning, Downdetector had logged more than eight million reports of disruptions affecting apps, games, and banking platforms.

 

Eight Sleep’s products rely on cloud connectivity to control temperature and track biometric data. When AWS went down, users lost access to the app that manages its water-cooled coils, leaving them stuck with whatever setting was last active.

 

Some beds overheated, others stopped cooling altogether, and several users said their devices became completely unresponsive....

 

 

 

[dmj]

https://aws.amazon.com/message/101925/

 

TLDR:  A major AWS outage in its US-EAST-1 region, which also affected Amazon.com, was caused by a DNS issue in the DynamoDB service. This problem cascaded, impacting other services like EC2, Lambda, and NLB. The root cause was a rare race condition in the DynamoDB DNS automation that resulted in an incorrect DNS record.

[Davit2005]

This is why I prefer as much home smart stuff to be local not relying on cloud at all. I use Home Assistant. The only issue I have is with WiFi smart plugs where I have not found anything as reliable as Hive, the three types I have tried mostly the relay fails or WiFi is PITA. I've tried Zigbee too and they fail. It must be the compact components?