Between Jan 28 5:45 am on Jan 28, 2019 and 1:01 am PST on Jan 29, 2019 Snowflake customers with deployments on AZURE - East US 2 and AZURE- West Europe regions were unable to retrieve or save worksheets in Snowflake UI.
As soon as Snowflake found the issue with the worksheets we debugged the problem and narrowed it down to failure of the CORS (Cross-Origin Resource Sharing) allowed headers check in the communication between the browser and Azure storage, where worksheet data is stored. Snowflake escalated the issue with Microsoft Azure and simultaneously worked on making a temporary mitigation to storage configuration to accept the headers as sent by the browser. Microsoft Azure fixed the problem in Azure Storage configuration and started to roll out the code at 7:43 PM Jan 28, 2019. The patch rollout was completed to both regions at 1:01 am Jan 29, 2019. Snowflake then confirmed the fix was complete and rolled back our temporary mitigation.
The connectivity disruption was caused by a bug in Microsoft Azure code where the CORS (Cross-Origin Resource Sharing) header “content-type” was rejected although “Content-Type” was listed as an allowed header. The Snowflake UI loaded in a web browser was sending the header tag as “content-type” (all lower case). Allowed headers should be, and always have been, matched case-insensitively, but Azure Storage bug began to match case-sensitively, causing all requests from browsers with “content-type” to be rejected as not matching CORS policy. This issue manifested in Chrome, Safari and Firefox. We are waiting for a full RCA from Microsoft Azure.
1. Snowflake Engineering already collects telemetry data from Snowflake UI to gain insights into the quality of service. We will continue to make changes in this area.
2. We are now waiting for Microsoft Azure team to publish an RCA for the incident and will follow up with them on future enhancements to prevent service interruptions due to these kind of errors.