We believe the underlying issue at hand to be fully resolved at this point.
We are currently working with Google Cloud Platform (GCP) on the root cause of this and Thursday's outages, and believe them to be strongly related to an upgrade of a networking component on GCP. This upgrade has completed, and in the short term we do not expect this issue to recur. Additionally, our team discovered various bugs within our resiliency system that made recovery on our end take longer than necessary, that we will be addressing in the coming week.
Our sincerest apologies for any inconvenience this issue has caused.
May 19, 06:12 PDT
We're continuing to work on mitigating and resolving issues that have stemmed from the networking issues.
May 19, 03:45 PDT
3 AM PDT: We are continuing to notice more network issues that are leading to service interruption.
May 19, 03:04 PDT
We continue to observe failures within our systems, that we're pretty sure is due to networking issues within Google Cloud. Our team is online and actively investigating as we attempt to mitigate these issues.
May 19, 02:46 PDT
We've identified the cause of this problem and we've implemented a fix.
May 19, 02:16 PDT
We're investigating reports of offline servers and slow connectivity.
May 19, 02:13 PDT