Some systems are experiencing major issues

About This Site

Real time service status from across the ATLAS Media Group portfolio

Past Incidents

18th September 2023

Queue & Content Processing Service Network Router Config Loss

We have identified an issue whereby our network routers are purging config on restarts which is not expected behaviour. We are working to restore the network now on our 3rd host.

  • Networking has now been restored. We will look into why restarted of the router is wiping config, as this is not intended behaviour and was not due to software upgrades.

    • 17th September 2023

    MastodonApp.UK Website Host Network Outage

    As part of emergent maintenance due to another ongoing issue, we have lost the ability to connect to the networking within one of our hosts, this is currently impacting both Universeodon and MastodonApp

  • Networking is now fully restored on the host that had failed. We will investigate exactly what caused this, however we have confirmed human error was partially to blame with regards to the delays in getting the service restored.

  • Networking has now been restored as far as we can see and we are monitoring to ensure all services come online fully.

  • It looks like there has been a partial corruption of configuration files within the router that is currently not coming online and connecting to the internet properly. We are continuing to look into this to see if we can get things back online ASAP.

    • Queue & Content Processing Service Universeodon - Content Processing Offline

    We have identified an issue with the content processing which is resulting in a full outage. New content is being queued but not processed at this time.

  • Service was previously restored. Historic update.

  • We managed to increase the resources allocated to the router and restore the networking issues that surfaced. We will resume work on scaling the service tomorrow.

  • We are going to need to increase the CPU capacity and memory capacity on the router within our 3rd dedicated host, this is going to involve partial outages to MastodonApp.UK (Loosing a single server though should not really be impacted due to non peak time) and Universeodon's ability to post, serve media and loss of 1 of the 2 web servers. This should take no longer than 15 mins this time.

  • We are continuing to see intermittent issues with the service. We had restored service earlier, however it appears to be non functional again. We are continuing to investigate.

    • 16th September 2023

    No incidents reported

    15th September 2023

    No incidents reported

    14th September 2023

    No incidents reported

    13th September 2023

    No incidents reported

    12th September 2023

    Queue & Content Processing Service Universeodon - Content Processing Delays

    We are seeing extended delays on the Universeodon content processing, specifically with regards to pushing content to other servers. We are investigating and working to increase capacity.

  • Service was restored.

    • MastodonApp.UK Website MastodonApp - Intermittent Outage

    We are seeing intermittent outages with the MastodonApp.UK site - We are investigating the root cause.

  • Service Was restored

    • 11th September 2023

    No incidents reported

    10th September 2023

    No incidents reported

    9th September 2023

    No incidents reported

    8th September 2023

    No incidents reported

    7th September 2023

    No incidents reported

    6th September 2023

    No incidents reported