Excel caused lost Covid-19 results

[mrstrong]

Well this is embarrassing for PHE

 

https://www.bbc.co.uk/news/technology-54423988

 

Absolutely gobsmacked to be honest.

 

Maybe the developers haven't heard of stack overflow ?

 

https://stackoverflow.com/questions/17684610/python-convert-csv-to-xlsx

Edited October 6, 2020 by mrstrong

[FN-GM]

Surely they should have backups of the original data and they could then import it again?

[synaesthesia]

Someone had a REAL spreadsheet fetish to want to do this. Seriously, in a field such as medicine, why would you use Excel as a data analysis tool? Yes it's powerful, and no-one would have batted an eye if they'd done it properly with the correct format but come on...

[Bumhug360]

Its likely they had a load of CSV files and needed to get the data out in a presentable format and chucked it on someones desk saying it needed to be done yesterday. Something quick and dirty was thrown together but it did the job. There is also a 99% chance that the person that made it emphasised that it was only a temporary solution and more work would need to be put into it but the higher ups just thought well it works so lets keep using it. I think most of us have been in the same situation where something we have fixed with blue tack and a bit of string has been deemed by management as a permanent fix and that money to do it properly would be made available when it stops working....

[MartinT]

It's not really Excel's fault, it has known and understood limits. These people were incompetent.

[TechMonkey]

What is even funnier is that the main issue was that they used XLS, not XLSX. Who even does that in this day and age!!

[FN-GM]

What is even funnier is that the main issue was that they used XLS, not XLSX. Who even does that in this day and age!!

 

I wondered that. Since Office 2007 - out of the box XLS is the default. 2 things come to mind.

 

1. Their computer system is set to save as XLS as default instead of XLSX. This was common when Office 2003 and below was still on production. Maybe it was never switched back or they still have Office 2003 and below in service?

 

2. They copied and pasted an old template and adapted it.

[MartinT]

What is even funnier is that the main issue was that they used XLS, not XLSX. Who even does that in this day and age!!

 

Agreed. Sometimes I send an XLSX out for amendment, and it comes back as an XLS. Sheesh!

[TechMonkey]

The only other option is that the export routine only supports XLS which I have spotted from some systems. This is normally down to they either haven't updated their export module to include XLSX or the developer hasn't looked at it yet so sticks with XLS because that will be fine.

[synaesthesia]

Understood it may be a "rush job" but we know better than most, sometimes you have to tell people to hold their horses if they want something doing properly and reliably - especially when it's vital information. ESPECIALLY when it's potentially life saving or threatening information.

[jthompson]

It's only been, what, 6 months since they knew they'd need to shift volumes of this stuff about. I may not know exactly how to do that, but am I wrong in being of the opinion that the scalability problem of IT is essentially a solved problem these days with cloud computing, containerisation, etc. Get somebody who's worked with Kubernetes or something. They'd probably being able to chuck something together in an afternoon that could ingest massive CSV files alllll day long without batting an eye.

[tech-man]

You couldn’t make this up!

[mikeprice]

Someone had a REAL spreadsheet fetish to want to do this. Seriously, in a field such as medicine, why would you use Excel as a data analysis tool? Yes it's powerful, and no-one would have batted an eye if they'd done it properly with the correct format but come on...

 

My wife used to work in a hospital - clinical audit dept

 

They spent most of their time analysing data and presenting the findings in order to improve the way things worked

 

she was one of the few in the whole places that used Access - nearly every doctor would use Excel for everything

[dhicks]

am I wrong in being of the opinion that the scalability problem of IT is essentially a solved problem these days with cloud computing, containerisation, etc.

 

The IT infrastructure is probably available, and there are probably plenty of individuals around with the skills and experience to produce a solution, but assembling those individuals and forming them into a team that can provide a specific solution to a given problem is going to take time. Hiring a developer and getting them up to speed on existing code and working practices would probably take at least 6 months normally. Parts of the system as explained in that BBC article make good sense - you want to farm out the work of gathering data to a bunch of disparate agencies, you need them to all be able to submit data in some kind of standard format, CSV files with defined headers is a reasonable choice.

 

Something like Python's Pandas library would read / write that amount of data with no problem, but somewhere upstream (by the sounds of it) you have a system which needs to import Excel files. It's even quite possible that the choice of "XLS" was simply a one-character typo - someone forgot the last "X" in a call to "DataFrame.to_excel(filename)" at the end of the script...

[mthomas08]

It's not really Excel's fault, it has known and understood limits. These people were incompetent.

 

Really? I would never guess

 

I remember people setup an access database then screamed when it would crash due to the limitations of it (think it was like 50 concurrent connections back in the day?).

 

Remember Windows XP and the NHS? exact same thing. It takes time, money and staff to upgrade these systems and help educate the users and tell them how to do things better.

 

Things like this will keep happening because... well... why would they listen to us... they just stick us on the list of 'outsourcing'.