As of approx 17:45 UTC+1 we were able to identify the issue as a memory leak within the content processing nodes which caused the node to crash. We have increased memory to allow the jobs to complete that were stuck and have rebooted the server. Full service has been restored since approx 18:00 UTC+1

We are aware of the extended nature of this outage and aim to restore service ASAP.

Due to existing maintenance and commitments we expect this issue to be ongoing for a little while longer. We expect to have the issue resolved fully by 19:30 UTC+1

We've identified an issue where all our content processing worker nodes have failed and a queue is forming of backlogged jobs.

Currently Universeodon is not properly federating as a result. We will resolve the issue as soon as we can.