All systems are operational.
The latest status updates from the team. Updates provided on known and ongoing incidents.
The main website for the ATLAS Media Group Team
The ATLAS Media Group Ltd. Blog Platform
Service which hosts static content across a number of our parked and inactive domains.
The primary billing panel for the Superior Networks clients.
The VPS Control panel for Superior Networks Customers.
The main Superior Networks website.
Our servers in the French location.
Our new London Region for Superior Networks
The mastodonapp.uk site
The database service storing appropriate content for the service
Handles the processing of content both internally to our server as well as communicating with remote servers
Storage for all videos, pictures and other content on the site
Used to handle delivery of e-mails to users
Search capability for MastodonApp.UK
The Universeodon public website
Universeodon's Database
The Universeodon Queue and content processing service.
Universeodon Media Storage
Universeodon's Advanced / full text search service.
Universeodon e-mail service
The ActivityPub relay hosted on relay.universeodon.com
Located in the London Region | Legacy Server
Located in the France Region
Located in the London Region
We will be updating MastodonAppUK to the latest version of Mastodon. During this time there will be a short outage while we restart the web server and run the relevant migrations.
We expect the maintenance overall to take approx 1 hour with an outage of approx 10 mins to complete the switch over.
We need to perform routine maintenance to our physical hardware located within the Redditch Data center. As part of this we will need to take all services offline for a short amount of time while we apply system and security updates to our shared storage pool and restart the underlying host. We will also be performing patching and upgrades to our application hosts during this time and will be failing over our applications across the nodes in the environment.
We expect the maintenance itself to take approx 3 hours with a short outage of up to 30 mins throughout this while we take the environment offline for full system restarts and hardware improvements.
We are now seeing service restored to normal usage. We believe this was a spike in content processing.
We believe the issue is caused by a spike in disk read delays on our database. We're working to identify the root cause.
We are seeing intermittent outages of the Mastodon website resulting in 502 errors being returned to users. Support are investigating.
No incidents reported.
No incidents reported.
No incidents reported.
No incidents reported.
We will be performing the version 4 upgrade to the MastodonApp.UK production website. We expect this maintenance to take up to 6 hours. During this time routine patching of the underlying hosts will also be applied. Updates will be posted on the following incident: https://status.atlas-media.co.uk/incidents/52
Service is fully operational and as such we're closing this tracking ticket.
We have had to reverse a change to our Queue service which as a result has overwhelmed our database service. We have reverted the change and are now waiting on the system to re-stabilize.
We have managed to identify and fix an issue with our content queuing setup, we believe this was causing the vast majority of the issues we were seeing including the lack of trending content being displayed. We will need to roll back one of the configuration changes, but will look to do so in the new year. We are continuing to monitor the instance.
There are currently significant delays to content processing and some content is getting stuck in our processing queues. Support are investigating.
We are also currently seeing issues with the queues not quite working as expected and in some cases notifications being delivered in the wrong / weird orders. We're working to get to the bottom of the issue.
We are seeing issues surface now that would indicate that either we're being given / using invalid data, or that the database schema did not fully transform as the software was supposed to. Our team are investigating now.
We are seeing some teething issues with the update, it looks like our trending posts and links have been lost / broken somehow as well as some of our server icon settings. Nothing major but we're keeping an eye and working to see if we can identify why this has happened.
Service has now been restored and the upgrade ran through without any real issues that we can currently see. The team will continue to monitor the environment for the next few hours to ensure all is well.
OS Upgrades on the two hosts which require updates to be applied during an outage have now been completed. We're now working to upgrade the Mastodon software to 4.0.2
We have now completed OS Upgrades on our public facing load balancer as well as our Redis instance. We're now working through upgrades on two of our four hosts which can only be upgraded with an outage. As soon as this is complete we will be running through the Mastodon Software upgrade.
Maintenance has now started and is actively under way. Please follow this incident as we provide updates.
We will be performing the version 4 upgrade to the MastodonApp.UK production website. We expect this maintenance to take up to 6 hours. During this time routine patching of the underlying hosts will also be applied.