Service has been restored, we have around 150k retry jobs which we hope are the majority of the jobs that were queued at the time of the outage but we know we have lost an unknown number of jobs we think mostly ingress jobs. We expect the site to be under extra pressure now for the next 12-24 hours while other servers on the fediverse re-try sending us their content and activity. We will close this incident now and pick up the existing main incident related to the original connectivity issues.

Our database server appears to have stopped routing traffic at around 1AM UK time, we are unsure as to why this happened and are aware it has happened previously. This has then taken the site down but also resulted in the loss of an unknown number of jobs to the server as our content processing continued to try to process and due to the length of time of the outage marked jobs as dead and not to be re-tried. It is impossible to know what jobs we lost or how many were lost, we are attempting to re-try the ones currently in the dead queue and have a huge backlog of retry jobs.

Universeodon has gone offline, we are investigating why.