Between the hours of 03:00 on March 10, 2020, PDT and 22:36 on March 12, 2020, PDT some Snowflake customers using Snowflake ODBC drivers in an embedded or standalone environment could not connect to Snowflake AZURE - US East 2 (Virginia).
The root cause of this issue was an outage in the AZURE Online Certificate Status Protocol (OCSP) responder service combined with customers using an older version of an ODBC driver in the range of 2.19.7 to 2.19.16.
Technical Details About Online Certificate Status Protocol (OCSP)
Snowflake drivers use OCSP to perform certificate revocation checks for SSL Certificates. The OCSP infrastructure also includes a Snowflake run OCSP Response Cache server that proactively fetches and caches OCSP responses for a predetermined set of URLs. This entire infrastructure, however, is dependent on the correct behavior of the OCSP responders. The drivers are designed with a Fail-Open mode to protect in the event the OCSP responder fails. A driver in fail-open mode overrides any failure to obtain a valid OCSP response and continue with the connection as opposed to the Fail Closed behavior where a connection is dropped.
1. Upgrade standalone drivers to use the latest driver 2.20.5. 2. Developed a workaround to replace the Snowflake ODBC driver bundled into Client Business Intelligence tools with the latest driver 2.20.5. 3. Worked with our business partners Microsoft Power BI, Tableau, QLIK Sense to make an emergency release of their online platforms to support the last few customers affected using the online solutions. 4. Updated OCSP Responder Cache Server maintained by Snowflake with failed OCSP URLs as an extended workaround while the drivers are being upgraded.
First, we apologize for the inconvenience caused by this incident. We have identified and planned the following improvements to our service and process to avoid such incidents in the future:
Azure OCSP Responder Service improvements
If you have any questions or issues, please send feedback to email@example.com or submit a support request ticket via the Snowflake Lodge Portal.
Note: The information contained in this report is confidential and is intended solely to promote safety and reduce customer risk.