Remember the Y2K bug? Microsoft confirms new Y2K22 issue (Exchange)

[elsiegee40]

https://news.sky.com/story/remember-the-y2k-bug-microsoft-confirms-new-y2k22-issue-12507401

 

Microsoft says it is aware of a programming flaw which saw some customers' Exchange servers stop processing emails just as the clock struck midnight on New Year's Eve.

 

System administrators, who are sharing workarounds on social media, have dubbed the bug Y2K22 - in the style of the Y2K bug which affected some computers at exactly the same time 22 years earlier.

 

Microsoft said its engineers had been "working around the clock on a fix" that wouldn't require customers to fiddle with their on-premise servers to get things moving again, but warned they found this "would require several days to develop and deploy".

 

Instead, those engineers are now working on a different update "which is in final test validation" that will require customer action, but also offer "the quickest time to resolution".

 

The technical issue seems to lie with the way that Microsoft was naming updates for its malware-scanning engine, putting the year, month, and day (220101) at the front of another four-digit number (0001).

 

Microsoft seems to use this system because when an update is named "2,201,010,001" it is simple to mathematically check which update is the most recent as it will have the higher value.

 

The problem appears to be that the field this number was stored in had a limit of being 31 bits, meaning the highest number that could be represented was 2,147,483,648 or 2 to the power of 31.

 

As soon as the clock ticked over to 2022, this naming system was going to exceed the maximum value that could be represented in 31 binary symbols.

 

The company hasn't yet confirmed the technical details, but its explanation seems to support the theory: "The version checking performed against the signature file is causing the malware engine to crash, resulting in messages being stuck in transport queues."

 

The software update affected is related to Microsoft's anti-malware scanning software, meaning that messages which should be queued up and checked are simply being queued.

 

One managed service provider warned on Microsoft's site that their company had seen a client queue 10,000 messages in less than 24 hours.

 

They said this risked filling up the server's storage and making it crash, potentially taking the business offline.

 

"Don't wait for the Microsoft patch if you are not sure your Exchange Server storage has the capacity to hold all queued messages without filling up disks and crashing," they wrote.

 

"Apply the workaround now to release the messages sooner than later."

 

The problem is the workaround involves disabling the malware filtering feature - potentially exposing companies to hackers.

 

Microsoft warns: "You should use one of these workarounds only if you have an existing malware scanner for email other than the engine in Exchange Server."

 

The Microsoft Exchange team said: "We expect to have this update to you shortly along with the actions required by you. We are sorry for any inconvenience that this issue has caused."

[southhamster]

How very amateur....

Being of a certain age I remember all the incredible effort (and lovely overtime!) that went into fixing Y2K potential issues. Everyone vowed that they would always write code that wouldn't get broken by date changes.

I wonder if the newer generations have forgotten/not been taught that hard lesson?

[mikeprice]

Ahh - yes Y2K

 

I got a nice big bonus every year for about 4 years because I was working on that.

 

Wonderful thing - al gave me loads of opportunity to set up networks of old computers in corners of computer rooms to test all sort of stuff

 

Still - you would have thought some companies would have learnt - or maybe not!

[elsiegee40]

How very amateur....

Being of a certain age I remember all the incredible effort (and lovely overtime!) that went into fixing Y2K potential issues. Everyone vowed that they would always write code that wouldn't get broken by date changes.

I wonder if the newer generations have forgotten/not been taught that hard lesson?

 

As another one who did very nicely out of Y2K, this is the Integer overflow problem rather than the date problem.

 

Still remembered from my PL/1 days:

 

Maximum number that can be stored in an integer (4 bytes) is 2,147,483,647 and smallint (2 bytes) is 32767.

 

So if your 10 digit number starts with the last 2 digits of the year, it was always going to be a problem when that year hit 22.

 

Basic coding cock up that is easily avoided.

 

Of course, just like the Y2K problem, the programmer undoubtedly thought this bit of code would be long gone before it became a problem

Edited January 3, 2022 by elsiegee40

[hardtailstar]

I was in year 7 in 2000.... But I remember all the buzz around it.

[FN-GM]

I was in year 7 in 2000.... But I remember all the buzz around it.

 

I was 10. I remember my great aunt stocking up on touches and batteries as she thought the power would go off!

[elsiegee40]

I was 10. I remember my great aunt stocking up on touches and batteries as she thought the power would go off!

 

It was a very real risk except there was an army of us, earning a small fortune, who managed to correct all the code that mattered.

 

And if you were on call on New Year’s Eve you could earn another small fortune. The employers got that one wrong. Techies wanted to be watching to see what what happened at midnight; they didn’t need to be paid more than usual to do their on call shift

[paulkerton]

I think people do forget the work that went into making sure Y2K wasn't the problem it could have easily been.

Some of the earliest paid work I had was to go from office to office and testing the machines in each with a floppy disk that checked whether the BIOS on the desktop machines were going to be a problem. I think we caught 1 that would be, but that was something that was running Windows 3.11 for Workgroups still!

[Davit2005]

Jus started helping to look after an engineering firms IT systems whilst still a mechanical engineer myself, over 5 years before I changed career to IT. 2K never effected us even though we had systems running NT3.5, NT4.0, Exchange 5.5 and WinFrame 1.7 for remote offices and mobile users, that brings back memories though.

[mikeprice]

At that time I was looking after a small comms system that had to run on DOS - no WIndows at all, of any kind

 

The PC support group (I only looked after bigger things at the time) wanted to 'upgrade' the computers this little system ran on - and gave me - over about 2 years - 3 different PCs to put it onto. None of them could run a driver for the serial comms port under DOS. I don't even know why I was supposed to find the driver - it wasn't my area at the time!

 

However, I eventually got them to give me another (yet another!) PC of a different make and found a driver that would work with it - and tested it and all that then implemented it.

 

The Instruction for all our systems were to shutdown and Power Off EVERYTHING - even the Main Frames were totally shut down - then the whole lot was powered up and restarted from the bottom up.

Somehow they managed to miss the PC running the backup system for the little comms package - the main system was fine.

 

So - the only Y2K problem that I actually saw was this system - but it only came to light some time in March when we needed to switch to the backup due to a power fault in the main Head Office computer room.

As a result every credit/debit card transaction for the whole company was rejected until I realised and got the operators to restart the PC and enter the date properly

The PC support people had forgotten to check the BIOS was Y"K compliant - and it just happened to be that one system they missed - and that one was the one system (probably) that didn't get powered off and restarted.

 

Naturally I got the blame - but I was used to that

[southhamster]

And another one!

 

https://www.mylondon.news/news/motoring/motoring-news/honda-owners-find-clocks-cars-22668370.amp

[elsiegee40]

And another one!

 

https://www.mylondon.news/news/motoring/motoring-news/honda-owners-find-clocks-cars-22668370.amp

 

I’m going to have to think about that one. What would make it “autocorrect“ in August?

[mikeprice]

My Honda is from that date so I was wondering

 

luckily it doesn't have the affected NAVI stuff - so it is OK

 

I don;t think it even knows the date!