We are sorry! Due to a combination of unlucky events, sync was failing for everyone across all apps and the billing dashboard. IT'S NOW FIXED!

TL;DR: we haven't been hacked, the service is safe, accepting or rejecting the certificate would both have been fine and not lead to any compromise of your data or issues. This is in addition to the fact that EteSync is end-to-end encrypted so even if we were hacked (we weren't!), it would not have mattered for the safety of your data.

What to do?

Just dismiss the notification if you got it, that's it. No need to accept or reject the certificate (though either are fine).

What happened?

The EteSync certificates have expired causing all HTTPS connections to fail. Since EteSync enforces HTTPS, everything was failing.

How it happened?

EteSync, like many services out there, uses the popular LetsEncrypt service for issuing TLS (HTTPS) certificates for its servers. LetsEncrypt certificates have a short validity period of three months by design. This means that services have to update their certificates at least once every three months.

EteSync follows best practices and has an automated process that runs nightly, checks the validity of all of the certificates and renews any certificates that are due to expire in the next month (so we have a safety buffer). Unfortunately, while updating something in the renewal process two months ago we introduced a bug that was causing renewals to fail. In addition, our error monitoring for this script was also failing (unknowingly to us). To make matters even worse it happened in the middle of the European night which meant it took us quite a few hours to notice it before it got fixed.

Short & long term fixes

We fixed the issue with the renewal script so the error itself is now fixed. We are also going to fix (edit: fixed!) the issue with the error reporting for this script to make sure that we get these errors a long time before the certificates fail.

Another contributing fact was that this process was running nightly. While this may sound like a good idea to run things when people are sleeping, it's actually not. First of all, EteSync has users from all over the world, so users are always awake. Second of all, this means that failure cases, like this one, happen at night for the developers. Which is definitely not what you want. We changed the automated process to happen during the day.

We are sorry!

You trust us to be operational non-stop, and we failed you. There's no excuse. Getting perfect 100% uptime is hard, though we've had a good run without any issues so far. Well, until today... We've learned from this error and are making sure it never happens again.