Users on-premises may be unable to send or receive email messages from on-premises or Exchange Online users

Incident
January 02, 2:45am AST

Users on-premises may be unable to send or receive email messages from on-premises or Exchange Online users

Status: Closed
Start: January 02, 1:54am AST
End: January 02, 2:45am AST
Duration: 50 minutes
Affected Components:
Update

January 02, 1:54am AST

January 02, 1:54am AST

Title: Users on-premises may be unable to send or receive email messages from on-premises or Exchange Online users
User Impact: Users on-premises may be unable to send or receive email messages from on-premises or Exchange Online users.
Current status: We're investigating a potential issue and checking for impact to your organization. We'll provide an update within 30 minutes.

Update

January 02, 2:19am AST

January 02, 2:19am AST

Current status: We’ve identified a latent code issue within Exchange Server 2016 and Exchange Server 2019 that results in a date check failure with the change of the new year for anti-malware updates starting with build version 2201010001. This date check failure will then cause mail-queuing for Exchange users hosted on-premises. We’ve deployed a new version in the *21 range to prevent any new customers from experiencing the issue and are working to finalize a fix for customers to implement that will restore mail flow for these users.
Scope of impact: This issue is specific to Exchange Online hybrid customers operating Exchange Server 2016 or Exchange Server 2019 for their on-premises users with anti-malware build 2201010001.
Start time: Saturday, January 1, 2022, at 12:00 AM UTC
Root cause: A latent version check code issue within Exchange Server 2016 and Exchange Server 2019 is causing the malware engine to crash when encountering anti-malware signature files within the *22 range. This in-turn leads to transport mail queueing for on-premises users within Exchange Online hybrid organizations.
Next update by: Sunday, January 2, 2022, at 9:30 AM UTC

Resolved

January 02, 2:45am AST

January 02, 2:45am AST

Title: Users on-premises may be unable to send or receive email messages from on-premises or Exchange Online users
User Impact: Users on-premises may be unable to send or receive email messages from on-premises or Exchange Online users.
More info: Please navigate to https://techcommunity.microsoft.com/t5/exchange-team-blog/email-stuck-in-transport-queues/ba-p/3049447 for instructions on implementing the fix for this problem.
Final status: We've finalized and implemented the fix for this issue. This fix requires additional customer actions and we've detailed these steps within the Exchange Team blog address provided above. Once impacted Exchange Online hybrid customers implement these actions, they will experience restored mail flow for their affected users.
Scope of impact: This issue is specific to Exchange Online hybrid customers operating Exchange Server 2016 or Exchange Server 2019 for their on-premises users with anti-malware build 2201010001.
Start time: Saturday, January 1, 2022, at 12:00 AM UTC
End time: Sunday, January 2, 2022, at 6:45 AM UTC
Root cause: A latent version check code issue within Exchange Server 2016 and Exchange Server 2019 caused the malware engine to crash when encountering anti-malware signature files within the *22 range. This in-turn lead to transport mail queueing for on-premises users within Exchange Online hybrid organizations.
Next steps: - We're working on a long-term fix to prevent future builds from encountering this issue.
This is the final update for the event.

Resolved

January 02, 2:45am AST

January 02, 2:45am AST

Resolved