Some systems are experiencing issues

About This Site

Real time service status from across the ATLAS Media Group portfolio

Past Incidents

13th September 2024

Queue Service MastodonApp.UK Content Processing Delays

We're seeing performance issues with our content processing service resulting in around a 1 hour delay on incoming posts. We're investigating the issue.

  • Changes have been updated on our configuration and infrastructure. Queue processing time is now back to normal.

    • 12th September 2024

    No incidents reported

    11th September 2024

    No incidents reported

    10th September 2024

    MastodonApp.UK & Universeodon.com & PixelFedapp.UK - Outages

    We are seeing outages across our compute cluster. We are unsure what the root cause of the issue is and are currently investigating.

  • We are now confident that full service has been restored.

  • I've made a further fix to our underlying infrastructure and believe I've managed to rectify the intermittent connectivity fault. I still need to resolve issues with a large number of members home timelines but that should be a lot smoother now this issue is resolved.

  • Universeodon is partially operational at this time. We are still seeing intermittent issues where the root cause remains unclear. Due to a series of other urgent issues that have come up I've not yet had time to fully troubleshoot the issues but will look to get this intermittent outage issue resolved ASAP.

  • Universeodon.com continues to have major disruption and ongoing issues. We have had to remove the corrupted redis database which has resulted in any sidekiq / content processing data being lost which we expect to be almost no data, however this has also resulted in some feeds such as home feeds being lost and needing re-creation. We're running into major issues when we try to regenerate the home feeds at this time and do not have capacity to continue working on the issue right now.

  • We have been able to fully restore MastodonApp.UK to normal operations. Unfortunately we're having much bigger issues when it comes to getting Universeodon.com operational due to a corrupted component (Redis) which is currently resulting in a full service outage while we look to minimise the data corruption issues there.

    • 9th September 2024

    No incidents reported

    8th September 2024

    No incidents reported

    7th September 2024

    No incidents reported

    6th September 2024

    No incidents reported

    5th September 2024

    No incidents reported

    4th September 2024

    No incidents reported

    3rd September 2024

    No incidents reported

    2nd September 2024

    No incidents reported

    1st September 2024

    Nexus Repository Server Nexus Service - Certificate Renewal Issue

    We have identified an issue whereby our Nexus server is failing to automatically renew certificates as expected, the site is currently offline as a result.

  • Service restored.

    • Multiple Services - Storage Failure

    We are investigating an issue impacting multiple services due to shared storage becoming unavailable. We will update as soon as we know more.

  • Full service has now been restored.

  • Storage has been restarted. Some services have come online however we are still trying to get all services online.

    • French Region Superior Networks - French Region Host Outage

    We are working to resolve issues with one of the nodes in our French region. We have started the process of moving clients onto a new operational node however our VPSCP is having issues with the migration. We have engaged our vendor that owns the VPSCP software and are awaiting further updates from them for us to resolve this issue.

  • We are back fully online.

  • We have identified the issue on both the old and new host and the original host is now fully operational. There was an additional issue when we attempted to migrate clients onto a new host, we are currently reversing the migrations now.

    • 31st August 2024

    Database [MAINTENANCE] Universeodon.com Database Migration

    Starting 19:30 BST we will commence migration of the Universeodon.com database server to new infrastructure to enable us to use additional capacity that has been provisioned.

  • The migration has fully completed and we're back up and running. We'll continue to keep an eye but all appears to be operational at this time.

  • The migration is now underway and we think it should be around 2 hours from now before full service is restored at the current transfer times.