Summary of the Clubhouse service disruption on June 1, 2017

On the afternoon of June 1, 2017, Clubhouse experienced a partial service outage that occurred between 2:07pm ET and 3:52pm ET.

Organizations that added or modified something in Clubhouse during that period of time may not have had their data recorded in our database. We have been able to recover all customer data from the affected period, and are in the process of communicating the recovered data to all affected customers.

The root cause of the issue was the interaction of two previously undiscovered bugs in Datomic, the database software that we use to store data on the backend, that caused our database to enter an unstable state.

We have been working with the Datomic team over the last week to address the issues at hand and they have released a new version of Datomic with fixes to the issues that we encountered (http://docs.datomic.com/release-notices.html).

Statement from the Datomic team:

"Release 0.9.5561.50 fixes a bug in the catalog that, in the unlikely circumstance where one has deleted a database and restored it from a backup without first having called gc-deleted-dbs, can cause a subsequent gc-deleted-dbs to delete that (active) database."

We have now deployed the updated version of Datomic and do not expect to encounter this problem again.

Every day, thousands of companies trust Clubhouse to keep mission-critical data safe and secure, and we take that responsibility very seriously. Now that the issue is resolved, we are planning a complete review of our operations and recovery procedures to ensure we’re taking every preventative measure we can to ensure the reliability and security of your data.

If you have any questions about the outage please don’t hesitate to contact us at support@clubhouse.io