[MAINTENANCE NOTIFICATION] MastodonApp.UK Version 4 Upgrade Thursday 22nd December 2022 10:18:04
We will be performing the version 4 upgrade to the MastodonApp.UK production website. We expect this maintenance to take up to 6 hours. During this time routine patching of the underlying hosts will also be applied.
Service is fully operational and as such we're closing this tracking ticket.
We have had to reverse a change to our Queue service which as a result has overwhelmed our database service. We have reverted the change and are now waiting on the system to re-stabilize.
We have managed to identify and fix an issue with our content queuing setup, we believe this was causing the vast majority of the issues we were seeing including the lack of trending content being displayed. We will need to roll back one of the configuration changes, but will look to do so in the new year. We are continuing to monitor the instance.
There are currently significant delays to content processing and some content is getting stuck in our processing queues. Support are investigating.
We are also currently seeing issues with the queues not quite working as expected and in some cases notifications being delivered in the wrong / weird orders. We're working to get to the bottom of the issue.
We are seeing issues surface now that would indicate that either we're being given / using invalid data, or that the database schema did not fully transform as the software was supposed to. Our team are investigating now.
We are seeing some teething issues with the update, it looks like our trending posts and links have been lost / broken somehow as well as some of our server icon settings. Nothing major but we're keeping an eye and working to see if we can identify why this has happened.
Service has now been restored and the upgrade ran through without any real issues that we can currently see. The team will continue to monitor the environment for the next few hours to ensure all is well.
OS Upgrades on the two hosts which require updates to be applied during an outage have now been completed. We're now working to upgrade the Mastodon software to 4.0.2
We have now completed OS Upgrades on our public facing load balancer as well as our Redis instance. We're now working through upgrades on two of our four hosts which can only be upgraded with an outage. As soon as this is complete we will be running through the Mastodon Software upgrade.
Maintenance has now started and is actively under way. Please follow this incident as we provide updates.
