We ran into an issue earlier today where our shared storage service failed on two of our servers. We're currently migrating the Database for MastodonApp.UK to a new server with more storage to prevent this issue from happening again in the future.
Storage has been recovered and is now operational. We will be migrating the outstanding workloads back to storage off of the shared pool we setup.
Universeodon should now be fully online. MastodonApp.UK continues to have issues due to the database being stopped without warning. We're looking into what's causing this issue.
We're seeing performance issues with our content processing service resulting in around a 1 hour delay on incoming posts. We're investigating the issue.
Changes have been updated on our configuration and infrastructure. Queue processing time is now back to normal.
We are seeing outages across our compute cluster. We are unsure what the root cause of the issue is and are currently investigating.
We are now confident that full service has been restored.
I've made a further fix to our underlying infrastructure and believe I've managed to rectify the intermittent connectivity fault. I still need to resolve issues with a large number of members home timelines but that should be a lot smoother now this issue is resolved.
Universeodon is partially operational at this time. We are still seeing intermittent issues where the root cause remains unclear. Due to a series of other urgent issues that have come up I've not yet had time to fully troubleshoot the issues but will look to get this intermittent outage issue resolved ASAP.
Universeodon.com continues to have major disruption and ongoing issues. We have had to remove the corrupted redis database which has resulted in any sidekiq / content processing data being lost which we expect to be almost no data, however this has also resulted in some feeds such as home feeds being lost and needing re-creation. We're running into major issues when we try to regenerate the home feeds at this time and do not have capacity to continue working on the issue right now.
We have been able to fully restore MastodonApp.UK to normal operations. Unfortunately we're having much bigger issues when it comes to getting Universeodon.com operational due to a corrupted component (Redis) which is currently resulting in a full service outage while we look to minimise the data corruption issues there.
