Some users may encounter delays or failures when opening links scanned by Safe Links
Some users may encounter delays or failures when opening links scanned by Safe Links
March 27, 11:27am ADT
March 27, 11:27am ADT
Title: Some users may encounter delays or failures when opening links scanned by Safe Links
User impact: Users may encounter delays or failures when opening links scanned by Safe Links.
Current status: We've identified unexpected latency in a section of infrastructure which facilitates the Safe Links feature. We've restarted a section of the affected infrastructure to mitigate impact, but a period of monitoring indicates this has not provided full relief. We're reviewing further telemetry and system logs to isolate the cause of the latency.
Scope of impact: Impact is specific to some users served through the affected infrastructure who are attempting to open links scanned by Safe Links.
Next update by: Monday, March 27, 2023, at 3:30 PM UTC
March 27, 12:25pm ADT
March 27, 12:25pm ADT
User impact: Users may encounter delays or failures when opening links scanned by Safe Links.
More info: Affected users may experience the following error 'We can't check the safety of this website right now.'
Current status: We've identified a potential root cause scenario which may be responsible for causing the unexpected latency within the affected infrastructure. We’re continuing to investigate telemetry and system logs further to verify and determine our remediation actions.
Scope of impact: Impact is specific to some users served through the affected infrastructure who are attempting to open links scanned by Safe Links.
Next update by: Monday, March 27, 2023, at 4:30 PM UTC
March 27, 1:13pm ADT
March 27, 1:13pm ADT
While our investigation continues, we're conducting additional targeted restarts within a section of the affected infrastructure and our telemetry is indicating that some customers may be experiencing a reduction or remediation in impact.
This quick update is designed to give the latest information on this issue.
March 27, 1:25pm ADT
March 27, 1:25pm ADT
User impact: Users may encounter delays or failures when opening links scanned by Safe Links.
More info: Affected users may experience the following error 'We can't check the safety of this website right now.'
Current status: We’re continuing to review telemetry and system logs to verify the identified potential root cause scenario. Additionally, we’re completing targeted restarts of a section of the affected infrastructure and this process is progressing as expected.
We've identified a potential root cause scenario which may be responsible for causing the unexpected latency within the affected infrastructure. We’re continuing to investigate telemetry and system logs further to verify.
Scope of impact: Impact is specific to some users served through the affected infrastructure who are attempting to open links scanned by Safe Links.
Next update by: Monday, March 27, 2023, at 5:30 PM UTC
March 27, 1:30pm ADT
March 27, 1:30pm ADT
Title: Some users may encounter delays or failures when opening links scanned by Safe Links
User impact: Users may have encountered delays or failures when opening links scanned by Safe Links.
More info: Affected users may have experienced the following error: 'We can't check the safety of this website right now.'
Final status: As our investigation into the underlying cause was ongoing, we confirmed through system telemetry that the restarts performed on the affected infrastructure successfully mitigated the issue, and that users are no longer experiencing impact when opening links.
Scope of impact: Impact was specific to some users served through the affected infrastructure who were attempting to open links scanned by Safe Links.
Start time: Sunday, March 26, 2023, at 9:50 PM UTC
End time: Monday, March 27, 2023, at 4:30 PM UTC
Preliminary root cause: Infrastructure responsible for handling Safe Links requests wasn't processing traffic as efficiently as expected.
Next steps:
- We're continuing to investigate what caused the infrastructure to operate inefficiently.
- We're monitoring system telemetry to confirm impact to end users does not recur.
We’ll publish a post-incident report within five business days.
March 27, 1:30pm ADT
March 27, 1:30pm ADT
Resolved
March 27, 2:17pm ADT
March 27, 2:17pm ADT
User impact: Users may encounter delays or failures when opening links scanned by Safe Links.
More info: Affected users may experience the following error: 'We can't check the safety of this website right now.'
Current status: Our analysis of telemetry and system logs thus far indicates that infrastructure responsible for handling Safe Links requests isn't processing traffic as efficiently as expected. As we investigate the underlying cause for the degradation, we continue to perform restarts in stages on the affected infrastructure in an attempt to mitigate the end-user impact.
Scope of impact: Impact is specific to some users served through the affected infrastructure who are attempting to open links scanned by Safe Links.
Preliminary root cause: Infrastructure responsible for handling Safe Links requests isn't processing traffic as efficiently as expected.
Next update by: Monday, March 27, 2023, at 7:00 PM UTC