ATLAS Media Group Status Page

We have confirmed that this was a blip in monitoring and there was no actual outage.

Our Redditch DC Networking device is again unreachable. We are investigating the issue at this time. This will be having a major impact to Universeodon.com

Service is fully operational and queues have cleared.

We are continuing to monitor the length of queues, we have just under 200k jobs currently outstanding with around a 40-50 min lag depending on the specific queue. I'm reluctant to push the processing any further than it already is as it may start to negatively impact the site. I will take another look in the morning and make sure everything has fully recovered as expected.

Our colo provider has been able to restart our networking gear and we've restarted our services to get the site online. We are now clearing through a backlog of content processing jobs which are coming in from while the site was offline so there might be some delays on content loading through, we will continue to monitor while queues are busy.

We are waiting on an update from the remote hands team at the site to confirm the status of our equipment. Once I have an update I will share it with you all.

Our primary router within our Redditch location appears to have gone offline. We are currently investigating the root cause of the issue. This will be having an impact to MastodonAppUK and Universeodon at this time.

Service has been restored, we have around 150k retry jobs which we hope are the majority of the jobs that were queued at the time of the outage but we know we have lost an unknown number of jobs we think mostly ingress jobs. We expect the site to be under extra pressure now for the next 12-24 hours while other servers on the fediverse re-try sending us their content and activity. We will close this incident now and pick up the existing main incident related to the original connectivity issues.

Our database server appears to have stopped routing traffic at around 1AM UK time, we are unsure as to why this happened and are aware it has happened previously. This has then taken the site down but also resulted in the loss of an unknown number of jobs to the server as our content processing continued to try to process and due to the length of time of the outage marked jobs as dead and not to be re-tried. It is impossible to know what jobs we lost or how many were lost, we are attempting to re-try the ones currently in the dead queue and have a huge backlog of retry jobs.

Universeodon has gone offline, we are investigating why.