MastodonApp.UK WebsiteIncreased 502 errors on MastodonApp.UK
We are seeing a series of 502 errors when loading MastodonApp.UK, some of the site will work fine whereas some areas appear to have issues. We are investigating and will look to identify which of the web servers is at fault.
We have identified an issue with our content processing which has resulted in a large number of images from remote servers not being processed as they should. This is due to the DNS settings on the server being lost. We are working to resolve the issue and put a permanent solution in place.
We have continued to monitor and the issue has been resolved.
We have managed to bring one of the two servers back online and work through the backlog of tasks to process. We are currently investigating why the other server is not resolving DNS properly.
We are seeing major issues with our content processing services. This started off with DNS resolution issues and at the current time the servers are not coming back online.
We have continued to see issues with DNS resolution and have now restored full capacity. We are now tracking this under the incident raised on Jan 6th.
Partial service is now restored. We are continuing to investigate and are now running at a reduced capacity.
Database ServiceMastodonApp UK Database Outage
Our database server host has ran out of disk space causing a major outage to our database server. We are working to bring services back online.
The data transfer has completed for our network infrastructure and full service has been restored. We will be looking to perform further maintenance behind the scenes to resolve some minor outstanding points.
We have completed the data transfer and as hoped have a significant amount more disk space available now on the host as a result of this migration back and forth. We will need to perform further emergency maintenance tomorrow to resolve some outstanding issues as well as some urgent maintenance to fully resolve this issue going forward and prevent it from hopefully happening again in the future.
The migration of the data to the new storage environment did succeed however the storage is not as fast or as reliable as it requires. This has however reduced the size of the disk to free-up enough capacity to move the server storage back without causing further outages.
We have provisioned additional infrastructure and started to migrate aspects onto this infrastructure. We are currently working to migrate the storage from our database server onto this infrastructure to enable us to complete a further migration in the coming days to resolve this issue on a longer term basis.
We are currently purchasing and deploying additional infrastructure to resolve this issue in the short term. We will need to work to validate that the approach scales in the longer term.
3rd January 2024
Database ServiceMastodonApp.UK Database Issues
We are seeing major performance issues with our database service which is causing partial outages to Mastodonapp.uk. We are working to identify and resolve the root cause.
This issue has been resolved.
We are monitoring the database service going forward, we know this issue is being caused by the high processing needs of our content processing service however we're unable to switch the processing onto a more efficient connection type due to a bug we have yet to identify. We hope that once we complete the software update to Mastodon this bug should be resolved.
