Some systems are experiencing issues

About This Site

Real time service status from across the ATLAS Media Group portfolio

Past Incidents

2nd October 2024

Queue Service MastodonApp Content Processing Failure

Around 8AM UK time today the content processing service for MastodonApp.UK ran out of application memory, this is a known issue and is made worse due to some capacity constraints I am currently working with. As of 16:00 content processing is back online and processing the backlog of jobs. We will continue to monitor.

  • Historically resolved.

    • 1st October 2024

    No incidents reported

    30th September 2024

    No incidents reported

    29th September 2024

    No incidents reported

    28th September 2024

    No incidents reported

    27th September 2024

    No incidents reported

    26th September 2024

    MastodonApp.UK & Universeodon Outages

    We have seen issues with MastodonAppUK and Universeodon.com which is currently impacting access to both sites. We are working to get things back online.

  • Both sites were fully online shortly after I posted this update, apologies for the slow status page update.

    • Queue Service MastodonApp.UK Content Processing

    Content processing for MastodonApp.UK appears to have failed as of around 11PM on the 25th Sept UK Time. We are working to get things back online.

  • The queue has been caught up and we're fully operational.

  • Content processing is now back online though running with a significant backlog. We're keeping an eye on the situation.

    • 25th September 2024

    No incidents reported

    24th September 2024

    Advanced Search Universeodon.com Full Text Search Outage

    We have identified an issue whereby the Universeodon.com full text / advanced search is offline and will need re-building. We will begin to work on this once the overall site stability has improved.

  • We are back fully online.

  • We're currently in the process of re-building all our search content, the vast majority of content should now be able to be searched over and search should mostly work.

    • 23rd September 2024

    Universeodon & MastodonApp Intermittent Outages

    We are continuing to see intermittent outages of both the Universeodon and MastodonApp web sites, we are investigating and working to get things back up and running.

  • We completed the deployment of a new physical host and have migrated our web servers onto this new host which appears to have resolved our intermittent outages.

  • MastodonApp.UK Content processing has now been migrated to a different server and is operational though working through a large backlog. An additional server is currently mid re-build and once complete will give us an additional web server for both sites which should both allow better performance in general but should also resolve some of the intermittent issues we've been seeing.

  • To try to reduce the impact of the issue we will be migrating the MastodonApp.UK Content processing worker onto a different host, the hope being this will free up capacity on the original host which is currently struggling with CPU capacity.

    • 22nd September 2024

    No incidents reported

    21st September 2024

    Storage Issues Impacting MastodonApp.UK

    We ran into an issue earlier today where our shared storage service failed on two of our servers. We're currently migrating the Database for MastodonApp.UK to a new server with more storage to prevent this issue from happening again in the future.

  • Storage has been recovered and is now operational. We will be migrating the outstanding workloads back to storage off of the shared pool we setup.

  • Universeodon should now be fully online. MastodonApp.UK continues to have issues due to the database being stopped without warning. We're looking into what's causing this issue.

    • 20th September 2024

    No incidents reported