URLs are unexpectedly being marked as malicious and removed from incoming Exchange Online email messages

Incident
December 17, 10:00pm AST

URLs are unexpectedly being marked as malicious and removed from incoming Exchange Online email messages

Status: Closed
Start: December 17, 2:24pm AST
End: December 17, 10:00pm AST
Duration: 7 hours 35 minutes
Affected Components:
Update

December 17, 2:24pm AST

December 17, 2:24pm AST

Title: URLs are unexpectedly being marked as malicious and removed from incoming Exchange Online email messages
User Impact: Users may notice that URLS are unexpectedly being marked as malicious and removed from incoming email messages.
Current status: We're investigating a potential issue and checking for impact to your organization. We'll provide an update within 30 minutes.

Update

December 17, 2:39pm AST

December 17, 2:39pm AST

Current status: We're analyzing example URLs that have been removed from the messages as well as the messages themselves to better understand the root cause of this issue. In parallel, we're investigating any recent changes to the service that generates and enforces these alerts to determine if they are contributing to the issue.
Scope of impact: Your organization is affected by this event, and users may notice that some URLs included in incoming messages through Exchange Online unexpectedly removed from the messages.
Next update by: Friday, December 17, 2021, at 8:00 PM UTC

Update

December 17, 3:51pm AST

December 17, 3:51pm AST

Current status: In addition to example URLS and messages, we've expanded our investigation into message headers to narrow down what could be leading to these URLs being marked as malicious.
Scope of impact: Your organization is affected by this event, and users may notice that some URLs included in incoming messages through Exchange Online unexpectedly removed from the messages.
Next update by: Friday, December 17, 2021, at 10:00 PM UTC

Update

December 17, 6:05pm AST

December 17, 6:05pm AST

Current status: We’ve determined that one of our subsystems had an outage which led data to be miscategorized and URLs to be incorrectly flagged as malicious. We've confirmed that messages are no longer being incorrectly quarantined. We're working on recovering messages that were incorrectly quarantined due to URLs being flagged as malicious.
Scope of impact: Any user may notice that some URLs included in incoming messages through Exchange Online are unexpectedly removed from the messages.
Start time: Friday, December 17, 2021, at 1:34 PM UTC
Root cause: One of our subsystems had an outage which led data to be miscategorized and URLs to be incorrectly flagged as malicious.
Next update by: Saturday, December 18, 2021, at 3:00 AM UTC

Resolved

December 17, 10:00pm AST

December 17, 10:00pm AST

Title: Users may notice that some incoming email that includes a URL is quarantined or routed to the Junk folder
User Impact: Users may have noticed that some incoming email that included a URL was quarantined or routed to the Junk folder.
Final status: After further review and discussion with affected users, we've clarified that affected messages were likely routed to a user's Junk folder or quarantined instead of arriving as expected. We've completed the email recovery operation on the unexpectedly quarantined messages, and we've confirmed that the messages are now recovering as expected to resolve this issue. These messages were resubmitted for expected evaluations so that they will again be correctly quarantined if other factors were present.
Scope of impact: Any user may have noticed that some incoming email that included a URL is quarantined or routed to the Junk folder.
Start time: Friday, December 17, 2021, at 1:34 PM UTC
End time: Saturday, December 18, 2021, at 2:00 AM UTC
Root cause: One of our subsystems had an outage which led data to be miscategorized and URLs to be incorrectly flagged as malicious.
Next steps: - We're continuing our investigation into the subsystem outage responsible for impact in order to harden our spam filtering codebase against this issue and prevent similar impact in the future.
This is the final update for the event.

Resolved

December 17, 10:00pm AST

December 17, 10:00pm AST

Resolved