Starting at 8/25/2021 14:04 UTC, three significant and simultaneous events occurred, leading to an authentication outage in [NA3.securid.com](http://na3.securid.com).

1. Our cloud provider reported a database outage lasting two minutes.
2. We found three previously unidentified defects in our failure handling.
3. A high volume of API traffic.

The three simultaneous issues consumed all available processing resources, which caused a unique failure in the authentication front-end that prevented it from accepting connections.

We attempted to resolve this issue according to best practice by performing rolling restarts of affected nodes. However, this did not resolve the issue because the restarted nodes were immediately overwhelmed.

We reverted software on the affected nodes to the July 2021 release, which allowed us to move the affected customers to a set of servers that was not impacted by these same issues. The August 2021 release did not introduce issues that caused the outage. It was the three previously mentioned issues happening simultaneously that caused the outage.

The following mitigations are being investigated to help avoid similar situations in the future:

• Fix three failure handling defects.
• Throttle unusual or incorrect admin API traffic.
• Review outage communications procedures and make necessary updates to help ensure more timely updates to our customers.

Thank you,
The SecurID SaaS Operations Team