Query Anomaly Detection Delayed
We found an issue with our anomaly detection process, that we've already fixed. New query anomalies will be detected normally. There is some additional processing that needs to be done to fill in events from the past few days; it should be done over the next few hours.
Sep 6, 14:51-20:15 UTC
Pipelines in sync. No data was lost. Service is operating normally.
Feb 25, 11:50-13:03 UTC
Metrics Pipeline Delayed
Our metrics pipeline has finished catching up, all metrics that were delayed in processing due to the failure are now available.
Feb 18, 09:48-12:37 UTC
Metrics ingest pipeline is falling behind
We tracked down the issue to a couple of corrupted partitions in the data ingestions pipeline. We were able to manually to advance them. No metrics data was lost and users will not see any gaps in their graphs.
Feb 16, 22:11 - Feb 17, 02:51 UTC
Backend server upgrade
We're completing the backend servers overhaul. No downtime or issues are expected.
Feb 14, 13:04 - Feb 15, 01:01 UTC
Internal Server Error in App
We've reset the login, so anonymous users can access now. We'll keep watch in case the limiting triggers again.
Feb 5, 00:46-01:17 UTC
Delayed Metrics Pipeline
We've ended processing the missing observations data. We'll continue to fix the missing node without effect on the application.
Dec 29, 21:57 - Dec 30, 03:52 UTC
Potential service impact
There's no impact to customers; the shard is healthy and only a standby server is having issues.
Dec 15, 17:15-17:20 UTC
Slow data processing (partial)
Querysample data ingestion is all caught up now. Another data ingestion process was saturating the network which caused querysample no to be able to catch up. We stopped the process that was causing network saturation and querysample was able to quickly process it's backlog of data.
Nov 26, 20:17 - Nov 27, 00:40 UTC
Gaps in Downsampled Data
Background processing has finished running without issue, customers that were affected should no longer see a gap in their data.
Nov 24, 17:58-18:43 UTC
Data Processing Behind
Metrics ingest is now functioning properly, no data was lost during the period of degraded performance.
Nov 23, 21:15-21:36 UTC