[MAINTENANCE UPDATE] This Incident is to provide live updates on ongoing maintenance.
We are planning to update the Mastodon software to the latest version. During this time the site will be unavailable and there may be delays in receiving remote posts and local posts being federated. We expect the maintenance to last no more than 2 hours.
Maintenance has concluded.
There is an outstanding task to resolve with content processing however at this time the temporary configuration can remain in place to be worked on at another time.
We have managed to bring our content processing back online temporarily to restore service. We are currently investigating what exactly went wrong.
The upgrades to our content processing appear to have resulted in some more significant issues and our content processing is currently unable to start. We are working to resolve the issue ASAP
All user facing upgrades have now completed. We will now be applying the patches to our content processing servers which may result in small delays with sending / receiving posts from remote servers.
We have completed the bulk of the Mastodon upgrade and are now running the latest version of Mastodon. We will now be rolling the update out on our second web server before progressing with further OS Patching to the physical servers which have patching outstanding.
We are comfortable at this point to progress with the upgrade to Mastodon itself. Maintenance should now conclude by 17:00 UK Time.
We are continuing to work towards progressing out upgrade. At this time we are awaiting database backups which failed to upload due to earlier server restarts. As soon as these are downloaded we will be able to progress with the Mastodon software upgrade.
We have identified a minor issue with our database server which we will need to resolve before we can fully progress with the upgrade. The maintenance is now likely to continue until 16:00 UK Time.
Upgrades to the host running the primary database is now well under way and we are performing one of two reboots to the database server. Our next step from here is to apply all of the OS Patches to the physical server that the database VM resides on and reboot everything at that point. From here we should be in a position to start the actual mastodon software update.
We are finishing the steps to upgrade our "Web02" VM (One of two user facing webservers that host the site) as well as the physical host that runs it and the router firmware on that host. Next we will be moving on to complete the same steps on the Database server before we start the actual Mastodon upgrade. During the database upgrades the site is likely to have intermittent outages.
We have now completed the most disruptive works to the site. We will now be moving on to a number of underlying OS Patches for the fleet where there will be less disruption.
Maintenance has now started. We will be starting to upgrade a number of components that are most disruptive first (Load Balancer, Redis Storage) and will then move to reduce resiliency on the site before upgrading the Mastodon software itself.
