Resolved - The root cause here was that one of our guild servers crashed unexpectedly and the automated recovery hit a bug we were fixing last week but haven't deployed the fix for yet. This issue stalled the recovery, leading to the extended degradation of service we saw here.

We will expedite deploying a fix for the recovery issue and will continue last week's investigation into the root cause that is leading a few of our systems to infrequently get into an escalating memory allocation spiral, ultimately leading to running out of memory.
Apr 23, 09:21 PDT

Resolved - Tonight's issue is over and Discord is working normally again. We're very sorry for the interruption this caused.

The root cause is under investigation. The team is actively working to understand the chain of events that led to this (and the last few) outages. We've got some good data on this crash and are putting together hypotheses that we can test in our staging & load test environments.

As mentioned in the last update, we do know the triggering event: a crash of one of our very large guilds that triggered a cascading failure in our sessions cluster. The crash is well understood and will not happen again, so we're confident we won't see a repeat of this cascading failure at this time.
Apr 19, 00:29 PDT

Resolved - We're not aware of any further issues at this time.
Apr 16, 18:01 PDT

Resolved - Discord is stable and we do not believe there will be a recurrence of today's issue at this time. We are still working on root causing what happened but are pushing out some mitigations to address some of the problems we saw in the incident.
Apr 12, 17:06 PDT