Tesla Posted November 14, 2016 Posted November 14, 2016 '186m needless emails': NHS-wide test message (and replies) crash system https://www.theguardian.com/society/2016/nov/14/186m-needless-emails-nhs-wide-test-message-and-replies-to-all-crash-system
Michael Posted November 14, 2016 Posted November 14, 2016 I'm guessing it's an internally hosted Exchange system, rather than O365. What I still can't get my head around is someone must have a created a Distribution List containing 1.2m users, but why would anyone want to do this? I can't think of any scenario why you'd want to do this for an organisation the size of the NHS!
Andrew_C Posted November 14, 2016 Posted November 14, 2016 Mrs_Cs ex works for NHS IT. I can believe it...
Oaktech Posted November 14, 2016 Posted November 14, 2016 I'm guessing it's an internally hosted Exchange system, rather than O365. It's currently a hybrid. It's a scaleout from an internal system that will eventually become an office365 deployment after it was decided that O365 could be made sufficiently secure. I've got a friend working on it - no he didn't send the email, but he knew (past tense) the person who did.
Andrew_C Posted November 14, 2016 Posted November 14, 2016 but he knew (past tense) the person who did.I suspect they have some very inventive ways of dealing with what might be considered "evidence".
zag Posted November 14, 2016 Posted November 14, 2016 I'm guessing it's an internally hosted Exchange system, rather than O365. What I still can't get my head around is someone must have a created a Distribution List containing 1.2m users, but why would anyone want to do this? I can't think of any scenario why you'd want to do this for an organisation the size of the NHS! I believe it was a test distribution list, and probably made by creating a rule rather than adding them individually Still utterly stupid though, especially as they seem to be running and old exchange server hybrid system, rather that full Office 365 like everyone else in the world now!
Blue_Cookeh Posted November 14, 2016 Posted November 14, 2016 (edited) Praise the lord for rules! I'm only sitting at 103 deleted e-mails so far today... I wonder how quickly Accenture managed to nip it in the bud. Can confirm normal work e-mail is going through relatively quickly. FYI for everyone else, we were all recently migrated to the NHSmail2 platform, which is built around Exchange 2013, I wouldn't call that ancient personally. Edited November 14, 2016 by Blue_Cookeh
Popular Post Joanne Posted November 14, 2016 Popular Post Posted November 14, 2016 I like this tweet https://twitter.com/martyn_amos/status/798125392284577793 5
timbo343 Posted November 14, 2016 Posted November 14, 2016 The other half is an employee of the NHS and low and behold she got an email about 9.30am with the subject test. Ive not looked at the headers yet. Since then emails have been delivered all over the place and shes got over 300 emails in her inbox. She normally doesnt get any.
Jamman960 Posted November 14, 2016 Posted November 14, 2016 My mrs is still having them come through, it must still be catching up as they say they've deleted the list, someone requested a read receipt on one message lol Good thing no one sent attachments in response
elsiegee40 Posted November 14, 2016 Posted November 14, 2016 Apparently the problem was compounded by a large number of users doing Reply All.
mikeprice Posted November 14, 2016 Posted November 14, 2016 Apparently the problem was compounded by a large number of users doing Reply All. Yup - there's all kinds of stupid out there
ellsandell Posted November 14, 2016 Posted November 14, 2016 Lots of people hitting reply-all... I am glad we have our own internal exchange, rather than some trusts who soley use NHS mail!
Michael Posted November 14, 2016 Posted November 14, 2016 Two things come to mind though - Hybrid setups are good for migrating on site Exchange servers to O365, but I wonder also if a full O365 deployment would have handled the load or not? I suspect the Reply All issue is because it's the default setting for O365... poor chap though, I'm sure he didn't mean to once he realised.
MatthewL Posted November 14, 2016 Posted November 14, 2016 So glad I no longer have a NHSFail account! It's all I've seen on Facebook today from former colleagues about this!
hardtailstar Posted November 15, 2016 Posted November 15, 2016 A friend of mine works in the NHS call center, she said that the email severely disrupted the whole call center!
FN-GM Posted November 15, 2016 Posted November 15, 2016 Yup - there's all kinds of stupid out there I think IT are to blame rather than the end users.
Blue_Cookeh Posted November 15, 2016 Posted November 15, 2016 I'm surprised it didn't happen sooner to be honest... after all, it is Accenture running that platform!
mikeprice Posted November 15, 2016 Posted November 15, 2016 I think IT are to blame rather than the end users. Oh yes - sound like some poor person in It that started it However - loads of people sending replies using reply/all was the part I was referring to - which shows how much they understand email!
strawberry Posted November 15, 2016 Posted November 15, 2016 I can't think of any scenario why you'd want to do this for an organisation the size of the NHS! Neither can I but I think we both know that senior management don't always indulge in the thinking when it comes to such matters. We have a massive amount of mailboxes that no one ever checks including.. Safeguarding,Danger,Alert,Forward planning, bullying..
Arthur Posted November 15, 2016 Posted November 15, 2016 (edited) I think IT are to blame rather than the end users. End users should get some of the blame when they start replying to the reply-all e-mails since it can prolong the email storm substantially. Obviously IT should get the main blame for setting up the distribution list in the first place. Edited November 15, 2016 by Arthur
Jamman960 Posted November 15, 2016 Posted November 15, 2016 I'd be interested to see a postmortem of this, my other half was still receiving replies that were obviously sent in the morning at midnight last night, outgoing emails were going straight out but incoming are severely delayed, an email sent at around 9pm arrived at 3am. I guess there isn't much they can do other than let it play out to avoid losing important mails.
Garacesh Posted November 15, 2016 Posted November 15, 2016 I'd be interested to see a postmortem of this, my other half was still receiving replies that were obviously sent in the morning at midnight last night, outgoing emails were going straight out but incoming are severely delayed, an email sent at around 9pm arrived at 3am. I guess there isn't much they can do other than let it play out to avoid losing important mails. Could put a rule in that any email with the subject "test" is dropped. Though, they'd have to be extremely precise about it, as to not snag "Blood test results" in the same net. Probably not worth it tbh, they just need to tell everybody to stop pressing Reply All. Preferably over the phone.
googlemad Posted November 15, 2016 Posted November 15, 2016 Has Microsoft / Google managed to implement a 'retract' feature yet for internal email (Well technically it isn't 'internal' these days but on the same domain account sort of thing) like Novell Groupwise used to offer? That was one of the biggest features absent when we moved to Google Mail! (The Google 'Undo' feature is pretty useless) Probably wouldn't work in this case mind!
zag Posted November 15, 2016 Posted November 15, 2016 Its actually easy to delete or block emails with certain subjects, or data in them. People say the NHS is the biggest employer in the world but its "only" got a million employees. Their server should still not have chocked. If they had gone full Office 365 then I doubt they would have had any problem at all, just need to jump on powershell and delete the message.
Recommended Posts
Create an account or sign in to comment
You need to be a member in order to leave a comment
Create an account
Sign up for a new account in our community. It's easy!
Register a new accountSign in
Already have an account? Sign in here.
Sign In Now