Resolved
Full recovery as of 12:24 PM UTC
Monitoring
We are seeing degraded performance again as latency to Google Cloud Storage. We are continuing to monitor and work with the GCP team.
Resolved
Full recovery as of 10:44am UTC.
We are investigating the root cause with the GCP team.
Monitoring
Substantial improvement starting 10:35am UTC. We are continuing to investigate and work directly with GCP to resolve this fully.
Identified
8:30am UTC - 9:25am UTC: Degraded performance on queries and writes in gcp-us-east4 from increased latency to Google Cloud Storage.
9:25am UTC: Escalated to partial outage both write and read paths due to climbing latency and timeouts from Google Cloud Storage.
* Queries may see elevated rates of 429s, and 500s.
* Writes may observe elevated rates of 408s, and 500s.
Our team is working to restore full system performance, and we are working with GCP to restore service as soon as possible.