Microsoft confirms Exchange Year 2022 problem
borncity.com
borncity.com
Compare the two snippets in [1]:
> Description: The FIP-FS "Microsoft" Scan Engine failed to load. PID: 23092, Error Code: 0x80004005. Error Description: Can't convert "2201010001" to long.
> Description: The FIP-FS Scan Process failed initialization. Error: 0x80004005. Error Details: Unspecified error.
I hope it's trend that continues because when it's going pear shaped every little morsel of information is important to narrow down the problem.
I'm less pleased they seemly didn't deploy this update to an internal on-prem test Exchange server before a wider release.
[1]: https://techcommunity.microsoft.com/t5/exchange-team-blog/em...
I think this means “when it’s going bad”, but what does “pear shaped” mean? And why a pear?
Is there actually an update here, or just some code that was probably sitting around for years until 2022 rolled around?
Antimalware signatures as code, anyone?
I can easily picture a scenario where there is pressure to reduce time to release signature updates, and someone looked at a process where there was early testing (that would have caught this) and said "Over the past 5 years, we've deployed hundreds of signatures with exactly zero failures; why are we still doing this process that adds a pointless delay?"
Or the Swiss railway axle counter [2].
[1]: https://en.wikipedia.org/wiki/Nuclear_Gandhi
[2]: https://www.reddit.com/r/programming/comments/4sco75/the_axl...
Another tale to tell your grandkids when they ask about those weird dates...?
> The newly updated scanning engine is fully supported by Microsoft. While we need to work on this sequence longer term, the scanning engine version was not rolled back, rather it was rolled forward into this new sequence. The scanning engine will continue to receive updates in this new sequence.[0]
[0] https://techcommunity.microsoft.com/t5/exchange-team-blog/em...
Of course, but in that case you would just store it as a normal date/timestamp value, e.g. ms or sec or heck even days since the epoch, truncated to days. Formatting it as a format string and then converting to an int is the part that's so weird.
What makes this weird is that, as hn_throwaway_99 notes, this isn't even a clever encoding. It's a stupid encoding that fails at the tasks of saving space or time. Both of those are accomplished better by the ordinary timestamp value that everything uses. Converting the timestamp to a string and then interpreting that string as an integer brings you:
- A less dense encoding: any value of a timestamp correctly represents a time, but not every value of an integer represents a date format string. You just introduced invalid dates into an encoding that had no reason to contain them.
- Equal sorting times: sorting integers is more or less the same no matter what the integers are.
- Vastly increased processing times: in the naive approach, you're handed a timestamp, which is an integer, and you use it directly. In this "optimized" approach, you're handed a timestamp, you convert it to a string, and then you convert the string back to another integer. This new integer is encoded so that you can't even mask out the bits you want, because the information you care about is stored in powers of 10 while the integer is represented in powers of 2. What did you gain?
However, it's very possible that this was written a decade ago, and even if someone did the example conversion at the time to 1,201,011,234 it wouldn't trigger that "magic number spidey-sense" like a number that's closer to 2.1 billion normally would.
In fact for all we know there's a comment next to it:
// TODO: convert to Int64 sometime before 2022Famous last words.
The problem with fixing it, is nobody thinks it will be their problem.
But.. they wouldn't be able to use int32 and therefore wouldn't have had this bug. :)
If you are making a calendar app, you want to store dates and not timestamps as, e.g. if I have a flight to another country then have an appointment at 9am, that 9am should be in the time zone of wherever I am physically present, which the calendar doesn't necessarily know.
Storing as an integer is compact and allows for easy comparison. It's a clever hack, but they should have used 64 bit, or at least a u32!
In 99% of the cases you only really want to involve timezones when presenting something to the user or taking input from them.
So you would create your calendar event by entering "9am, timezone X", and the app would most probably just apply that offset and store that as a timestamp. But if it's an all-day event... I should probably take a look at android's calendar content provider.
Referring to something like "Christmas day" as timezoneless doesn't help either. Anyone who celebrates it celebrates it in a timezone. Christmas Day EST is a different event than Christmas Day CEST. Trying to represent them as the same event will only lead to heartbreak.
Generally I think Dates (without time) are usually considered time-less and therefore timezone-less.
In fact the examples you mention are only a concern when considering the start or end of Christmas Day, and not the actual date of Christmas Day.
I kinda feel you've never worked with a large code base or database that has a legacy going back 25+ years...older than many of the commentators on this thread.
Not all of the data types used might have been obvious in their "danger", especially in ancient code. Sure you could run some analysis over the code, but can that analyser determine from a #def or variable name the true intent of the value being stored?
I wrote an application for a customer in 1986, and stone me, they were still running it in a DOSBOX emulator in 2004. Lucky for me dates weren't important.
DNS serial numbers are 32 bit unsigned and the convention is to use YYYYMMDDNN
So this _convention_ breaks in year 4295, or if someone forgot to put unsigned 2148. It is only a convention though DNS only cares that the serial increases.
Because with four-digit years this kind of date would never have fit into a signed 32-bit integer. Incidentally, this shows that the timestamp was taken into use after Y2K, since none of the pre-millennium years would have fit.
Maybe this code is dozens of years old and no one thought it would be in production but there should have been a test case for this at the very least.
Meanwhile I was denied a job at Microsoft out of college almost 30 years ago because I didn’t remember how to divide coins properly to find the fake coin.
No, the result is a parser failure. That’s what the error message[1] says, at least.
But, the Y2K issue, back then, was in 1000s of software from 1000s of vendors. The only Y2K affect I remember was the first issue of 2600 in 2000.
Microsoft Exchange stops passing mail due to bug on 1/1/22 (677 points / 355 comments)
If you're still using integers (for ids, timestamps, etc) then just go with 64-bits. It avoids any potential problem with data size in the future, and if the size stays small then it won't matter anyway. Storage is cheap and CPU cache lines are 64-bit now too.
I wonder if there were detectable drops in internet traffic due to fewer emails?
The malware scanning service of MS Exchange crashes, because it treats a yyMMddHHmm timestamp as a signed integer when verifying a signature file.
Turns out that 2201010001 is negative when treated as a 32 bit integer (the greatest positive one is 2147483647, and 2021 had fewer than 47 months).
I can only assume that somebody wrote that "timestamp string as integer" code, checked that it worked correctly (at the time) and then just assumed they must be good on data type range.
The anti malware releases probably started out with definition version 1, 2, 3, and so on, until someone complained that version 37363 doesn't mean much. Some smart guy probably realised that as long as the numbers increased, they could put anything in there, so the 1201010000 version was born, incremented a day at a time.
As MXToolbox states:
> The serial number is an unsigned 32 bit value assigned to your SOA record must be between 1 and 4294967295. […] It has become common to set your serial number with a date format to make it easier to to manage. This format uses 10 digits to represent the date and then a two digit sequence number with the format of YYYYmmddss.
Do any SOA query against hosted DNS and you'll probably see exactly this pattern appear.
The fact they used a long go store the version also suggested that someone may not have realised that longs are the same size as its, depending on your platform and compiler. The effective distinction between the two in Microsoft's compiler ended in the 32 bit era, so using a long to store anything might be seen as a red flag.
Based on the fix linked, the current antimalware release seems to be based on December 33rd, 2021. That will buy them enough time to write a patch that just uses long longs for these timestamps, or if their customers don't like installing updates they might just go back to incrementing the number by one at some point.
> Addendum: A fix ist available.
Are we speaking German now? Or is that a typo for isn’t? Or is, maybe?
(because you will always miss something)