Real time service status from across the ATLAS Media Group portfolio
No incidents reported
At approx 2AM this morning our gateway device which provides internet access to all of our services stopped connecting back to the internet. We are waiting now for on-site techs to reboot the device. We are unsure as to what caused this as it's a freshly replaced device.
As of last night we have identified and resolved the fault that was causing instability on our network device as well as performance issues. We're now seeing noticeably better performance on Universeodon and our content processing is also running notably faster.
We have been able to restart the network device once again and it so far appears to be running stable. We have moved our Universeodon Redis and SideKiq content processing onto a different host within the cluster which has some slightly different physical networking properties which we hope will make it easier to deal with any future disruptions until the physical networking for the second app server is updated. Universeodon is currently online though we have a lot of content that will need processing as servers continue to retry sending us their events and updates. Apologies for the disruption, we are now awaiting a diagnostics update from Ubiquti who is the vendor for our networking hardware to identify why this has happened.
The team at the DC have rebooted the device for us, unfortunately it went back offline fairly quickly. We are investigating what's causing this behaviour.
We are about to start performing essential maintenance to our on-site infrastructure. During this time Universeodon will be fully offline and MastodonAppUK will be partially unavailable. We expect the outage to last no more than 60 mins.
No incidents reported
We are currently moving the Superior Networks VPSCP to a new underlying host. There may be a short period where the VPS Control Panel is unavailable while we complete the switch.
There will be no impact to customer VPS's running during this time.
No incidents reported
We're opening this incident to track and keep you updated on the maintenance we will be performing throughout the day. We expect and plan for Universeodon.com to complete the maintenance a fair bit earlier than MastodonAppUK, this is in part to minimise both sites being down allowing us to focus on one at a time and partially because MastodonAppUK has more assets to migrate to the new infrastructure.
You can follow this incident to get updated on when we're taking sites down and our progress overall throughout the day.
The cluster network issues were resolved under a different incident. As the migration going forward will be involving near zero downtime (Planned) we will spin up separate incidents in each case where we are going to potentially need to take the site offline for a short window.
Yesterday we completed the move to retire one of our legacy cluster servers, this means MastodonAppUK is currently running at reduced resiliency and reduced capacity for the website. Universeodon has all services currently operating.
Unfortunately the MastodonAppUK migration was just taking too long and we don't have capacity to support the migration running into the early hours of the morning. We've aborted the migration and will also scrub down and reset the new server ready to attempt a different solution tomorrow at some point.
The Universeodon DB Migration has catastrophically failed again and as a result we've restored the site using the legacy infrastructure for the time being. We will look to perform the migration now tomorrow afternoon / evening UK time and will look to use an alternative way of operating the migration. We're seeing extensive performance issues on Universeodon due to heavy latency between our legacy cluster and new cluster.
The MastodonAppUK DB Migration appears to be running fine at this point, subject to it completing in the near future we can switch over to using the new DB tonight with further maintenance now required also tomorrow.
I've done some digging into the Universeodon DB Issues, it looks like when it previously failed it didn't re-start where it left off and is re-migrating from scratch and will then purge all of the original records once the new records are written, in this case our statuses table (Which is over 150GB in size) is having some weirdness. I'm going to let it run for a bit longer but if it isn't complete within the next 1 hour we will have to abort the migration and re-start everything on the old DB before purging what we tried to copy across and re-running the migration tomorrow.
I'm back and checking everything now. The Universeodon migration appears to still be happening though it is reporting a larger DB now on the new server than on the old so it's likely something has gone pretty significantly wrong there. The MastodonAppUK migration appears to still be underway and slowly progressing.
Unfortunately the Universeodon database migration is taking considerably longer to complete compared to what we expected and is not going to be complete before I need to go offline for the evening.
With that in mind I'm going to re-start the MastodonApp migration and the work will continue later this evening. I'm hoping everything will be online before the end of today.
We have stopped the MastodonAppUK DB Migration to free up additional network capacity and IO capacity for the Universeodon transfer as I have a hard stop and have plans this evening so won't be around for a few hours. I'm hoping the remaining 50-60GB of Universeodon data will transfer fairly quickly and I can get the site online before I need to leave this evening and MastodonApp will then resume the transfer once Universeodon's is complete with the goal to bring the site back online tonight.
The Universeodon transfer is continuing and appears to be around 3/4 of the way there now. I'm hopeful it will complete in the next hour or so and enable us to bring the site back online.
MastodonAppUK's transfer is going slower than we would like and is likely to now result in the site being offline until later tonight due to availability issues this afternoon.
The new MastodonAppUK DB Server is now setup and we're going to take the site offline now to enable the migration to the new environment.
We're continuing to monitor the progress on the Universeodon one, it appears to have picked up where it left off as far as we can tell.
Good Morning, I'm re-starting the maintenance now.
The Universeodon transfer appears to have hung around 5:40 this morning (UK Time). I'm hopeful I can re-start it where we left off. I will keep you updated.
Once I'm happy the Universeodon transfer is working I'll switch focus to the MastodonAppUK one.
It looks like the Universeodon DB Migration is due to take another 5-10 hours (It's hard to tell) given the volume sizes I can see and how much has transfered so far. With that in mind we're going to have to leave the site off overnight tonight and I'll look to get everything operational first thing in the morning once I have woken up as it's now 1AM here and I do need sleep! Apologies for the extended outage this very much was not the intention.
MastodonAppUK is now back online and we're going to pause our work for the site there.
Universeodon is still mid-migration so we're going to wait for that to complete and finish the switch-over before calling it a night.
With how long the migrations and upgrades are taking (And how far over the maintenance window we already are) once the MastodonAppUK Upgrade completes we will bring the site back online and pause the work there migrating the DB to the new server tomorrow.
Universeodon is still in the process of migrating once that's complete we will get configurations updated and the site online tonight which should conclude the disruptive maintenance to the site for the time being.
The Universeodon transfer is now underway and I'm finishing the configuration on the new DB Server ready to start to switch the site over to it. I'm expecting there to be some hickups while we switch over but the priority is going to be to get the web tier online first and then the content processing afterwards. There are a reasonable number of configuration files I need to update across the servers once we're moved over.
The MastodonAppUK Upgrade to Postgres 17 is running and once complete I'll bring the site up again to validate all is working before repeating the transfer that's currently underway for Universeodon.
The MastodonAppUK Backup has now completed and the upgrade from 16.x to 17.x is now underway, this is likely to take 2-3 hours from what we've seen.
The new DB Server for Universeodon is now ready and we're going to take the site offline to facilitate the data migration, we expect this also is likely to take between 1 and 3 hours after which we will switch all traffic to our new cluster where we expect performance to be noticeably better.
Universeodon's DB upgrade has now completed, we're getting the site back online now to make sure everything is working. We'll keep the site up for a short period to ensure everything is stable and then we'll look to take it offline one final time to migrate the DB onto our new cluster.
MastodonAppUK's DB is currently backing up and then we will start the 16.x to 17.x upgrade process.
The Universeodon Postgres update continues.
We're taking MastodonAppUK back offline now to perform the 16.x to 17.x upgrade on Postgres before we then move it onto the new cluster.
MastodonApp's back-end database has now been upgraded to Postgresql 16.x. All services have now been restarted and we're letting everything run for a little bit now to make sure everything is performing as expected.
Universeodon is currently progressing through the 16.x to 17.x upgrade, once that is complete we will get the site online to test before we perform the migration of the database onto our new server cluster. Unfortunately this means the Universeodon maintenance is running over schedule and we don't currently have a firm estimated completion time.
The MastodonApp Database has finished it's backup and is now in the process of upgrading from 15.x to 16.x
We are going to stop the Universeodon services now and get the Universeodon database moved from 16.x to 17.x ready for the final part of the maintenance to move it to the new compute cluster.
Universeodon is currently operational on the upgraded Postgres version. We're going to leave the site online for a short while longer while we perform backups for MastodonAppUK's Database, once MastodonAppUK's DB starts it's 15.x to 16.x upgrade we will take Universeodon back offline and prepare it for the 16.x to 17.x upgrade.
Universeodon's upgrade of Postgres from 15 to 16 should be complete, we are brining the site online now and letting things run to test then we will finish the cleanup of the 15 install before moving from 16 to 17.
We're about to start the same process now on MastodonAppUK which will result in some intermittent outages followed by a somewhat extended outage while we perform the upgrade.
The Universeodon database is currently in the process of upgrading from 15.x to 16.x, it may take a little while due to the size of our database and the historic content we store being well federated. We will look to bring the site back online, ensure everything is running smoothly and then we will repeat the process to upgrade from 16 to 17.
The first stage of the Universeodon maintenance has gone well. We now need to take the site offline again and will be performing an upgrade of our Postgresql instance from 15 to 16 before we bring the site back online to verify everything is operational. We will need to repeat this afterwards for 16 to 17.
We are starting some work now to Universeodon. We're taking the content processing and websites offline to perform upgrades to the underlying database. We will then bring the site back online for a short amount of time while we complete some additional work ready to move the database to the new environment. Once we're happy that we're ready to move we will update again and take the site offline while the database is moved.
We are performing maintenance to MastodonAppUK now to upgrade to the latest version of Mastodon (4.4). We expect there to be a short outage while we restart servers at around 20:30 UK time (UTC +1) but otherwise maintenance will progress without impact to the community.
Maintenance fully completed.
The upgrade has now mostly completed. We're running through the last of the database migrations in the background but the site itself should now show as the latest version of Mastodon. We will continue to keep an eye for the next couple of hours to make sure everything is behaving as expected.
Due to a major email campaign currently being executed for Universeodon.com we are hitting our per-second limit of sending e-mails across the system. This is meaning there's likely to be a 30-40 min delay on e-mails being sent while we process all of the e-mails backlogged to be delivered to registered users on Universeodon.
We will be upgrading the Universeodon site to the current 4.4 version of Mastodon this evening starting at 7PM UK Time (GMT +1). During this time the site may be offline for short periods of time while we apply the upgrade steps. Further updates will be posted here during the course of the maintenance.
All maintenance is fully complete and the site is working as expected.
The site itself is now up to date, we're applying the last of the database migrations behind the scenes before we push some updates to search and our content processing queues as a new queue has been introduced which is not currently configured. We will update you as we progress or if there is any impact to service.
Maintenance has started, it looks like the majority of the work for the upgrade can be done behind the scenes and shouldn't result in an outage. We're running into a small problem at the moment which we weren't expecting but expect to be upgraded shortly.
Our primary router within our Redditch location appears to have gone offline. We are currently investigating the root cause of the issue. This will be having an impact to MastodonAppUK and Universeodon at this time.
Service is fully operational and queues have cleared.
We are continuing to monitor the length of queues, we have just under 200k jobs currently outstanding with around a 40-50 min lag depending on the specific queue. I'm reluctant to push the processing any further than it already is as it may start to negatively impact the site. I will take another look in the morning and make sure everything has fully recovered as expected.
Our colo provider has been able to restart our networking gear and we've restarted our services to get the site online. We are now clearing through a backlog of content processing jobs which are coming in from while the site was offline so there might be some delays on content loading through, we will continue to monitor while queues are busy.
We are waiting on an update from the remote hands team at the site to confirm the status of our equipment. Once I have an update I will share it with you all.
No incidents reported
No incidents reported