Some systems are experiencing issues.
The latest status updates from the team. Updates provided on known and ongoing incidents.
The main website for the ATLAS Media Group Team
The ATLAS Media Group Ltd. Blog Platform
Service which hosts static content across a number of our parked and inactive domains.
The primary billing panel for the Superior Networks clients.
The VPS Control panel for Superior Networks Customers.
The main Superior Networks website.
Our servers in the French location.
Our new London Region for Superior Networks
The mastodonapp.uk site
The database service storing appropriate content for the service
Handles the processing of content both internally to our server as well as communicating with remote servers
Storage for all videos, pictures and other content on the site
Used to handle delivery of e-mails to users
Search capability for MastodonApp.UK
The Universeodon public website
Universeodon's Database
The Universeodon Queue and content processing service.
Universeodon Media Storage
Universeodon's Advanced / full text search service.
Universeodon e-mail service
The ActivityPub relay hosted on relay.universeodon.com
Located in the London Region | Legacy Server
Located in the France Region
Located in the London Region
We have confirmed that this was a blip in monitoring and there was no actual outage.
Our Redditch DC Networking device is again unreachable. We are investigating the issue at this time. This will be having a major impact to Universeodon.com
Service is fully operational and queues have cleared.
We are continuing to monitor the length of queues, we have just under 200k jobs currently outstanding with around a 40-50 min lag depending on the specific queue. I'm reluctant to push the processing any further than it already is as it may start to negatively impact the site. I will take another look in the morning and make sure everything has fully recovered as expected.
Our colo provider has been able to restart our networking gear and we've restarted our services to get the site online. We are now clearing through a backlog of content processing jobs which are coming in from while the site was offline so there might be some delays on content loading through, we will continue to monitor while queues are busy.
We are waiting on an update from the remote hands team at the site to confirm the status of our equipment. Once I have an update I will share it with you all.
Our primary router within our Redditch location appears to have gone offline. We are currently investigating the root cause of the issue. This will be having an impact to MastodonAppUK and Universeodon at this time.
No incidents reported.
No incidents reported.
No incidents reported.
No incidents reported.
Service has been restored, we have around 150k retry jobs which we hope are the majority of the jobs that were queued at the time of the outage but we know we have lost an unknown number of jobs we think mostly ingress jobs. We expect the site to be under extra pressure now for the next 12-24 hours while other servers on the fediverse re-try sending us their content and activity. We will close this incident now and pick up the existing main incident related to the original connectivity issues.
Our database server appears to have stopped routing traffic at around 1AM UK time, we are unsure as to why this happened and are aware it has happened previously. This has then taken the site down but also resulted in the loss of an unknown number of jobs to the server as our content processing continued to try to process and due to the length of time of the outage marked jobs as dead and not to be re-tried. It is impossible to know what jobs we lost or how many were lost, we are attempting to re-try the ones currently in the dead queue and have a huge backlog of retry jobs.
Universeodon has gone offline, we are investigating why.