Past Incidents

Service is fully operational.

A fix has now been rolled out to move the intensive read/write operations from the server running our primary database server onto some of our other infrastructure. We're confident this should resolve the poor performance that has been observed and will be monitoring over the coming 24 hours.

Apologies for the late update on the last comment, we incorrectly posted it on a previous incident. We have one final attempt we will make tonight to restore full performance to the environment and should this not work we will look to roll back the production impacting changes and get some sleep.

We are actively trying to push out a new infrastructure component to take the load off our primary database node, however data replication has been failing and we're working to identify the root cause of this new issue.

We are currently working to deploy a database replica on a new physical host which will allow us to move the cause of the heavy IO Usage onto that node and away from our current primary database node. Once complete we should start to see the site stabilize.

We are now seeing service restored to normal usage. We believe this was a spike in content processing.

We believe the issue is caused by a spike in disk read delays on our database. We're working to identify the root cause.

Service is fully operational and as such we're closing this tracking ticket.

We have had to reverse a change to our Queue service which as a result has overwhelmed our database service. We have reverted the change and are now waiting on the system to re-stabilize.

We have managed to identify and fix an issue with our content queuing setup, we believe this was causing the vast majority of the issues we were seeing including the lack of trending content being displayed. We will need to roll back one of the configuration changes, but will look to do so in the new year. We are continuing to monitor the instance.

There are currently significant delays to content processing and some content is getting stuck in our processing queues. Support are investigating.

We are also currently seeing issues with the queues not quite working as expected and in some cases notifications being delivered in the wrong / weird orders. We're working to get to the bottom of the issue.

We are seeing issues surface now that would indicate that either we're being given / using invalid data, or that the database schema did not fully transform as the software was supposed to. Our team are investigating now.

We are seeing some teething issues with the update, it looks like our trending posts and links have been lost / broken somehow as well as some of our server icon settings. Nothing major but we're keeping an eye and working to see if we can identify why this has happened.

Service has now been restored and the upgrade ran through without any real issues that we can currently see. The team will continue to monitor the environment for the next few hours to ensure all is well.

OS Upgrades on the two hosts which require updates to be applied during an outage have now been completed. We're now working to upgrade the Mastodon software to 4.0.2

We have now completed OS Upgrades on our public facing load balancer as well as our Redis instance. We're now working through upgrades on two of our four hosts which can only be upgraded with an outage. As soon as this is complete we will be running through the Mastodon Software upgrade.

Maintenance has now started and is actively under way. Please follow this incident as we provide updates.