Unfortunately Threadloop was unavailable for more than 8 hours (I want to say "tonight", but that won't be the case for everyone). The outage happened just after Guro and I went to bed, so we didn't notice it until we woke up this morning (what a thing to wake up to!). 

The issue is fixed now, and all it took was a server restart. After a little investigating, the issue happened because of a maintenance system that queried the database for data from the last 24 hours. This system was part of a third-party library I installed recently to get more performance metrics about Threadloop (which requests are slow, and so on). I'm not sure exactly what happened, but in essense this maintenance system stopped our database from running, and never enabled it again. Either the system ran out of memory working through a large amount of data, or there was some other issue that prevented it from starting the database server again.

The lessons I'm taking away from this is to not trust third-party libraries to the degree that I did. I had only ran this system once in production, but not with as much data as it was fed after a full day of operation. I should have also made sure to be available (and not asleep) when it was running. I have now disabled it from running again, because I don't trust it anymore. I will also look at our server configuration to make sure our database restarts if it is ever shut down again. Hopefully this should be enough to prevent this from happening again.

I'm very sorry that Threadloop was unavailable for such a long time.

- Endre