There is ongoing disruption to both Universeodon and MastodonApp.UK and our team are working to resolve these issues.
MastodonApp.UK is currently disrupted due to semi-planned maintenance to resolve issues with our underlying hosts and capacity which is currently constrained and requires re-work to resolve, we are actively working to resolve these capacity issues and bring the service fully online. Universeodon.com has been accidentality impacted due to issues with the sites content processing configuration and a lack of content processing capacity. We're currently running at a significantly degraded level of capacity on the Universeodon processing capacity. We are working as quickly as possible to bring everything online.
From what we can now see the service is stable and the site is operating as expected. Our improved monitoring will continue to look and detect any issues with the service as we move forward.
We continue to monitor the situation, we did have an issue where our content processing for MastodonApp.UK fully locked up and froze.
We will plan to do the other essential maintenance now tomorrow evening as it's unlikely we will see sufficient service stability to want to start further works tonight.
We have now restored full service on all our sites and are continuing to monitor.
We will shortly need to perform further disruptive maintenance in order to resolve a critical issue however expect the disruption to be significantly shorter this time.
We're now seeing major disruption across all sites due to an issue with the host running our underlying database servers. We believe this may be due to the increased load due to MastodonApp content processing catching up but is impacting MastodonApp.UK, Universeodon.com and Pixelfedapp.uk
We have now restored full service, it appears our final issue resided with our Redis environment and for some reason we ran into an ongoing issue where some containers are unable to access the network until the core router on the server has been restarted, we're unsure as to why this happens but is a known issue with our current infrastructure deployment.
We're continuing to monitor the site and errors at this time, our external monitoring is showing green across the board and we will continue to observe the situation to ensure there are no further residual issues.
Universeodon content processing should now be back to capacity, we need to revert some of the manual changes we made before we call that complete.
MastodonApp.UK continues to experience issues, while the data migration is complete we're running into major issues restoring access to our web tier, we're working to resolve the issue at this time.
The Universeodon relay is now back online and serving requests. We are still working to restore access to MastodonApp.UK
We are also currently seeing issues again with content processing on Universeodon.com, we are working to resolve these issues at this time.
Content processing for Universeodon.com has now caught up fully and we're continuing to monitor for any errors while the data migration continues to restore normal capabilities.
The MastodonApp.UK site and Universeodon relay continue to be offline as the data migration is taking considerably longer than expected. We are hopeful that once one of the 4 processes running to move the required data onto their new respective servers completes that we will start to see a performance improvement and the others will be quick to follow. Apologies for the continued outage which we understand is not ideal.
We have started to scale up content processing for Universeodon.com on one of our servers which has previously had issues, we are monitoring the failures closely to ensure there are no spikes in failures that may cause data loss or disruption.
The MastodonAppUK Outage and Universeodon relay outage is taking considerably longer for the data migration to complete than we had hoped, we are continuing to monitor and will restore service as soon as we can.
