[Maintenance] Server Cluster Migration (Impacting All Federated Services) Saturday 27th September 2025 11:16:10

Yesterday we completed the move to retire one of our legacy cluster servers, this means MastodonAppUK is currently running at reduced resiliency and reduced capacity for the website. Universeodon has all services currently operating.

Unfortunately the MastodonAppUK migration was just taking too long and we don't have capacity to support the migration running into the early hours of the morning. We've aborted the migration and will also scrub down and reset the new server ready to attempt a different solution tomorrow at some point.

The Universeodon DB Migration has catastrophically failed again and as a result we've restored the site using the legacy infrastructure for the time being. We will look to perform the migration now tomorrow afternoon / evening UK time and will look to use an alternative way of operating the migration. We're seeing extensive performance issues on Universeodon due to heavy latency between our legacy cluster and new cluster.

The MastodonAppUK DB Migration appears to be running fine at this point, subject to it completing in the near future we can switch over to using the new DB tonight with further maintenance now required also tomorrow.

I've done some digging into the Universeodon DB Issues, it looks like when it previously failed it didn't re-start where it left off and is re-migrating from scratch and will then purge all of the original records once the new records are written, in this case our statuses table (Which is over 150GB in size) is having some weirdness. I'm going to let it run for a bit longer but if it isn't complete within the next 1 hour we will have to abort the migration and re-start everything on the old DB before purging what we tried to copy across and re-running the migration tomorrow.

I'm back and checking everything now. The Universeodon migration appears to still be happening though it is reporting a larger DB now on the new server than on the old so it's likely something has gone pretty significantly wrong there. The MastodonAppUK migration appears to still be underway and slowly progressing.

Unfortunately the Universeodon database migration is taking considerably longer to complete compared to what we expected and is not going to be complete before I need to go offline for the evening.

With that in mind I'm going to re-start the MastodonApp migration and the work will continue later this evening. I'm hoping everything will be online before the end of today.

We have stopped the MastodonAppUK DB Migration to free up additional network capacity and IO capacity for the Universeodon transfer as I have a hard stop and have plans this evening so won't be around for a few hours. I'm hoping the remaining 50-60GB of Universeodon data will transfer fairly quickly and I can get the site online before I need to leave this evening and MastodonApp will then resume the transfer once Universeodon's is complete with the goal to bring the site back online tonight.

The Universeodon transfer is continuing and appears to be around 3/4 of the way there now. I'm hopeful it will complete in the next hour or so and enable us to bring the site back online.

MastodonAppUK's transfer is going slower than we would like and is likely to now result in the site being offline until later tonight due to availability issues this afternoon.

The new MastodonAppUK DB Server is now setup and we're going to take the site offline now to enable the migration to the new environment.

We're continuing to monitor the progress on the Universeodon one, it appears to have picked up where it left off as far as we can tell.

Good Morning, I'm re-starting the maintenance now.

The Universeodon transfer appears to have hung around 5:40 this morning (UK Time). I'm hopeful I can re-start it where we left off. I will keep you updated.

Once I'm happy the Universeodon transfer is working I'll switch focus to the MastodonAppUK one.

It looks like the Universeodon DB Migration is due to take another 5-10 hours (It's hard to tell) given the volume sizes I can see and how much has transfered so far. With that in mind we're going to have to leave the site off overnight tonight and I'll look to get everything operational first thing in the morning once I have woken up as it's now 1AM here and I do need sleep! Apologies for the extended outage this very much was not the intention.

MastodonAppUK is now back online and we're going to pause our work for the site there.

Universeodon is still mid-migration so we're going to wait for that to complete and finish the switch-over before calling it a night.

With how long the migrations and upgrades are taking (And how far over the maintenance window we already are) once the MastodonAppUK Upgrade completes we will bring the site back online and pause the work there migrating the DB to the new server tomorrow.

Universeodon is still in the process of migrating once that's complete we will get configurations updated and the site online tonight which should conclude the disruptive maintenance to the site for the time being.

The Universeodon transfer is now underway and I'm finishing the configuration on the new DB Server ready to start to switch the site over to it. I'm expecting there to be some hickups while we switch over but the priority is going to be to get the web tier online first and then the content processing afterwards. There are a reasonable number of configuration files I need to update across the servers once we're moved over.

The MastodonAppUK Upgrade to Postgres 17 is running and once complete I'll bring the site up again to validate all is working before repeating the transfer that's currently underway for Universeodon.

The MastodonAppUK Backup has now completed and the upgrade from 16.x to 17.x is now underway, this is likely to take 2-3 hours from what we've seen.

The new DB Server for Universeodon is now ready and we're going to take the site offline to facilitate the data migration, we expect this also is likely to take between 1 and 3 hours after which we will switch all traffic to our new cluster where we expect performance to be noticeably better.

Universeodon's DB upgrade has now completed, we're getting the site back online now to make sure everything is working. We'll keep the site up for a short period to ensure everything is stable and then we'll look to take it offline one final time to migrate the DB onto our new cluster.

MastodonAppUK's DB is currently backing up and then we will start the 16.x to 17.x upgrade process.

The Universeodon Postgres update continues.

We're taking MastodonAppUK back offline now to perform the 16.x to 17.x upgrade on Postgres before we then move it onto the new cluster.

MastodonApp's back-end database has now been upgraded to Postgresql 16.x. All services have now been restarted and we're letting everything run for a little bit now to make sure everything is performing as expected.

Universeodon is currently progressing through the 16.x to 17.x upgrade, once that is complete we will get the site online to test before we perform the migration of the database onto our new server cluster. Unfortunately this means the Universeodon maintenance is running over schedule and we don't currently have a firm estimated completion time.

The MastodonApp Database has finished it's backup and is now in the process of upgrading from 15.x to 16.x

We are going to stop the Universeodon services now and get the Universeodon database moved from 16.x to 17.x ready for the final part of the maintenance to move it to the new compute cluster.

Universeodon is currently operational on the upgraded Postgres version. We're going to leave the site online for a short while longer while we perform backups for MastodonAppUK's Database, once MastodonAppUK's DB starts it's 15.x to 16.x upgrade we will take Universeodon back offline and prepare it for the 16.x to 17.x upgrade.

Universeodon's upgrade of Postgres from 15 to 16 should be complete, we are brining the site online now and letting things run to test then we will finish the cleanup of the 15 install before moving from 16 to 17.

We're about to start the same process now on MastodonAppUK which will result in some intermittent outages followed by a somewhat extended outage while we perform the upgrade.

The Universeodon database is currently in the process of upgrading from 15.x to 16.x, it may take a little while due to the size of our database and the historic content we store being well federated. We will look to bring the site back online, ensure everything is running smoothly and then we will repeat the process to upgrade from 16 to 17.

The first stage of the Universeodon maintenance has gone well. We now need to take the site offline again and will be performing an upgrade of our Postgresql instance from 15 to 16 before we bring the site back online to verify everything is operational. We will need to repeat this afterwards for 16 to 17.

We are starting some work now to Universeodon. We're taking the content processing and websites offline to perform upgrades to the underlying database. We will then bring the site back online for a short amount of time while we complete some additional work ready to move the database to the new environment. Once we're happy that we're ready to move we will update again and take the site offline while the database is moved.