13 Deployments Affected: MI-20220525
Incident Report for Snowflake
Postmortem

Dear Customer,

Snowflake Engineering has completed the postmortem of this service incident. A detailed Root Cause Analysis (RCA) is available on the Snowflake Community site.

https://community.snowflake.com/s/article/MI-20220525

We apologize for the service incident and the impact on your applications.

If you have any questions or difficulty accessing the Snowflake community site, please send feedback by submitting a support case or calling Snowflake Technical Support.

Posted Jun 03, 2022 - 12:31 PDT

Resolved
Current status: We've implemented the fix for this issue and monitored the environment to confirm that service was restored. If you experience additional issues or have questions, please open a support case via Snowflake Community.

Customer experience: Customers hosted in the specified regions may have experienced delays or failures while performing Snowpipe data ingestion workflows that require the copy_history function. If the timespan set for the copy_history query is between May 9, 2022, and May 25, 2022, at 06:30 PT, then your queries may continue to be impacted. If the timespan is set for a period of time that does not query copy_history prior to when the mitigation was implemented at 06:30 PT, then your queries are no longer impacted.

Workaround: As a workaround, you can alter the timespan of your queries to a shorter time frame (e.g., one or two hours).

Incident start time: 03:37 PT May 25, 2022

Incident end time: 06:30 PT May 25, 2022

A root cause analysis (RCA) document will be published within seven business days.
Posted May 25, 2022 - 09:47 PDT
Monitoring
Current status: We've implemented the fix for this issue, and we'll continue to monitor the environment until we're confident all services are functioning properly.

Customer experience: Customers hosted in the specified regions may have experienced delays or failures while performing Snowpipe data ingestion workflows that require the copy_history function. If the timespan set for the copy_history query is between May 9, 2022, and May 25, 2022, at 06:30 PT, then your queries may continue to be impacted. If the timespan is set for a period of time that does not query copy_history prior to when the mitigation was implemented at 06:30 PT, then your queries are no longer impacted.

Workaround: As a workaround, you can alter the timespan of your queries to a shorter time frame (e.g., one or two hours).

Incident start time: 03:37 PT May 25, 2022
Posted May 25, 2022 - 08:46 PDT
Identified
Current status: The investigation determined that a recent configuration update caused the issue. We implemented a solution at approximately 06:30 PT; however, Snowpipe queries involving copy_history may continue to be impacted if you are using a timespan that queries copy_history prior to the mitigation at 06:30 PT. We're monitoring the environment to confirm recovery, and we're investigating any potential options for resolving issues with copy_history query timespans that involve time ranges prior to 06:30 PT.

Customer experience: Customers hosted in the specified regions may experience delays or failures while performing Snowpipe data ingestion workflows that require the copy_history function. If the timespan set for the copy_history query is between May 9, 2022, and May 25, 2022, at 06:30 PT, then your queries may continue to be impacted. If the timespan is set for a period of time that does not query copy_history prior to when the mitigation was implemented at 06:30 PT, then your queries are no longer impacted.

Workaround: As a workaround, you can alter the timespan of your queries to a shorter time frame (e.g., one or two hours).

Incident start time: 03:37 PT May 25, 2022
Posted May 25, 2022 - 07:41 PDT
Investigating
Current status: We're investigating an issue with Snowpipe (Data Ingestion). We'll provide an update within 60 minutes.

Customer experience: Customers hosted in the specified regions may experience delays while attempting to execute queries. Affected customers may encounter query failures.

Incident start time: 03:37 PT May 25, 2022
Posted May 25, 2022 - 06:35 PDT
This incident affected: GCP - Europe West 2 (London) (Snowpipe (Data Ingestion)), Azure - West US 2 (Washington) (Snowpipe (Data Ingestion)), Azure - West Europe (Netherlands) (Snowpipe (Data Ingestion)), Azure - Australia East (New South Wales) (Snowpipe (Data Ingestion)), AWS - Asia Pacific (Singapore) (Snowpipe (Data Ingestion)), AWS - Asia Pacific (Tokyo) (Snowpipe (Data Ingestion)), AWS - Canada (Central) (Snowpipe (Data Ingestion)), AWS - Asia Pacific (Seoul) (Snowpipe (Data Ingestion)), AWS - US East (N. Virginia) (Snowpipe (Data Ingestion)), GCP - US Central 1 (Iowa) (Snowpipe (Data Ingestion)), GCP - Europe West 4 (Eemshaven) (Snowpipe (Data Ingestion)), AWS - US West (Oregon) (Snowpipe (Data Ingestion)), and AWS - Asia Pacific (Sydney) (Snowpipe (Data Ingestion)).