Increased API Latency & Failed Message Sends

Incident Report for Discord

Resolved

The secondary that previously failed has successfully snuggled up near a warm fire with a cozy blanket and some hot cocoa and is now adequately warm to serve traffic at nominal latency.

Posted Oct 04, 2018 - 11:35 PDT

Monitoring

The secondary appears to be sufficiently warm now, and we are noticing error rates and latency drop to nominal levels.

Posted Oct 04, 2018 - 11:31 PDT

Investigating

We are aware of issues regarding higher than normal latency, and failed message sends. The team is aware of the root cause of the issue (a host error on one of our database secondaries caused it to reboot).

During this time, the secondary is "cold" as it loads data into memory. We are monitoring the instance as it warms up, and expect latency to fully recover in 5-15 minutes, and latency to continue to as the secondary gradually warms up.

Posted Oct 04, 2018 - 11:28 PDT

This incident affected: API.