AWS - US West (Oregon): INC0096454
Incident
Report for Snowflake
Postmortem
Snowflake Engineering has completed the postmortem of this service incident. A detailed Root Cause Analysis (RCA) is available on the Snowflake Community site:
https://community.snowflake.com/s/article/INC0096454
Resolved
Current status: We've implemented the fix for this issue and monitored the environment to confirm that service was restored. If you experience additional issues or have questions, please open a support case via Snowflake Community.
Customer experience: Affected users would have experienced Snowpipe ingestion latency and delays when streaming data via the Kafka Connector while the backlog of ingestion requests is processed.
Incident start time: 18:00 UTC November 07, 2023
Incident end time: 04:30 UTC November 08, 2023
Preliminary root cause: A system configuration change resulted in Snowpipe ingestion performance degradation.
A root cause analysis (RCA) document will be published within seven business days.
Customer experience: Affected users would have experienced Snowpipe ingestion latency and delays when streaming data via the Kafka Connector while the backlog of ingestion requests is processed.
Incident start time: 18:00 UTC November 07, 2023
Incident end time: 04:30 UTC November 08, 2023
Preliminary root cause: A system configuration change resulted in Snowpipe ingestion performance degradation.
A root cause analysis (RCA) document will be published within seven business days.
Monitoring
Current status: We've temporarily rolled back specific Snowflake software configurations to the previous release which addressed the data ingestion performance and we'll continue to monitor the environment until we're confident all services are functioning properly.
Customer experience: Affected users may experience Snowpipe ingestion latency and delays when streaming data via the Kafka Connector while the backlog of ingestion requests is processed.
Incident start time: 18:00 UTC November 07, 2023
Incident end time: 04:30 UTC November 08, 2023
Preliminary root cause: A system configuration change resulted in Snowpipe ingestion performance degradation.
Customer experience: Affected users may experience Snowpipe ingestion latency and delays when streaming data via the Kafka Connector while the backlog of ingestion requests is processed.
Incident start time: 18:00 UTC November 07, 2023
Incident end time: 04:30 UTC November 08, 2023
Preliminary root cause: A system configuration change resulted in Snowpipe ingestion performance degradation.
Update
Current status: We've temporarily rolled back specific Snowflake software configurations to the previous release to tackle the data ingestion performance problem. We're actively monitoring the service for complete recovery. We'll provide another update within 2 hours.
Customer experience: Affected users may experience Snowpipe ingestion latency and delays when streaming data via the Kafka Connector while the backlog of ingestion requests is processed.
Incident start time: 18:00 UTC November 07, 2023
Customer experience: Affected users may experience Snowpipe ingestion latency and delays when streaming data via the Kafka Connector while the backlog of ingestion requests is processed.
Incident start time: 18:00 UTC November 07, 2023
Update
Current status: We're temporarily reverting a subset of Snowflake software configurations to its previous release to address the data ingestion performance issue. We'll provide another update within 60 minutes.
Customer experience: Affected users may experience Snowpipe ingestion latency and delays when streaming data via the Kafka Connector while the backlog of ingestion requests is processed.
Incident start time: 18:00 UTC November 07, 2023
Customer experience: Affected users may experience Snowpipe ingestion latency and delays when streaming data via the Kafka Connector while the backlog of ingestion requests is processed.
Incident start time: 18:00 UTC November 07, 2023
Update
Current status: We've identified the source of the issue, and we're developing and implementing a fix to restore service. We'll provide another update within 60 minutes.
Customer experience: Affected users may experience Snowpipe ingestion latency and delays when streaming data via the Kafka Connector while the backlog of ingestion requests is processed.
Incident start time: 18:00 UTC November 07, 2023
Customer experience: Affected users may experience Snowpipe ingestion latency and delays when streaming data via the Kafka Connector while the backlog of ingestion requests is processed.
Incident start time: 18:00 UTC November 07, 2023
Update
Current status: Our telemetry indicates that the backlog of requests continues to process as expected. We'll provide an update within 2 hours or upon complete service recovery.
Customer experience: Affected users may experience Snowpipe ingestion latency and delays when streaming data via the Kafka Connector while the backlog of ingestion requests is processed.
Incident start time: 18:00 UTC November 07, 2023
Customer experience: Affected users may experience Snowpipe ingestion latency and delays when streaming data via the Kafka Connector while the backlog of ingestion requests is processed.
Incident start time: 18:00 UTC November 07, 2023
Update
Current status: Our monitoring indicates that the rollback was successful in mitigating the issue; however, affected customers may still experience delays while the backlog of ingestion requests is processed. We'll provide an update within 60 minutes.
Customer experience: Affected users may experience Snowpipe ingestion latency and delays when streaming data via the Kafka Connector.
Incident start time: 18:00 UTC November 07, 2023
Customer experience: Affected users may experience Snowpipe ingestion latency and delays when streaming data via the Kafka Connector.
Incident start time: 18:00 UTC November 07, 2023
Identified
Current status: We've completed a rollback of a recent configuration change, and our telemetry indicates that service health is recovering. In parallel, we're continuing to monitor and investigate further to isolate and address the source of the issue. We'll provide an update within 60 minutes.
Customer experience: Affected users may experience Snowpipe ingestion latency and delays when streaming data via the Kafka Connector.
Incident start time: 18:00 UTC November 07, 2023
Customer experience: Affected users may experience Snowpipe ingestion latency and delays when streaming data via the Kafka Connector.
Incident start time: 18:00 UTC November 07, 2023
Investigating
Current status: We're investigating an issue with Snowpipe ingestion delays. We'll provide an update within 60 minutes.
Customer experience: Affected users may experience Snowpipe ingestion latency.
Incident start time: 18:00 UTC November 07, 2023
Customer experience: Affected users may experience Snowpipe ingestion latency.
Incident start time: 18:00 UTC November 07, 2023
This incident affected: AWS - US West (Oregon) (Snowpipe (Data Ingestion)).