We are seeing major issues with our content processing services. This started off with DNS resolution issues and at the current time the servers are not coming back online.
We have continued to see issues with DNS resolution and have now restored full capacity. We are now tracking this under the incident raised on Jan 6th.
Partial service is now restored. We are continuing to investigate and are now running at a reduced capacity.
Database ServiceMastodonApp UK Database Outage
Our database server host has ran out of disk space causing a major outage to our database server. We are working to bring services back online.
The data transfer has completed for our network infrastructure and full service has been restored. We will be looking to perform further maintenance behind the scenes to resolve some minor outstanding points.
We have completed the data transfer and as hoped have a significant amount more disk space available now on the host as a result of this migration back and forth. We will need to perform further emergency maintenance tomorrow to resolve some outstanding issues as well as some urgent maintenance to fully resolve this issue going forward and prevent it from hopefully happening again in the future.
The migration of the data to the new storage environment did succeed however the storage is not as fast or as reliable as it requires. This has however reduced the size of the disk to free-up enough capacity to move the server storage back without causing further outages.
We have provisioned additional infrastructure and started to migrate aspects onto this infrastructure. We are currently working to migrate the storage from our database server onto this infrastructure to enable us to complete a further migration in the coming days to resolve this issue on a longer term basis.
We are currently purchasing and deploying additional infrastructure to resolve this issue in the short term. We will need to work to validate that the approach scales in the longer term.
3rd January 2024
Database ServiceMastodonApp.UK Database Issues
We are seeing major performance issues with our database service which is causing partial outages to Mastodonapp.uk. We are working to identify and resolve the root cause.
This issue has been resolved.
We are monitoring the database service going forward, we know this issue is being caused by the high processing needs of our content processing service however we're unable to switch the processing onto a more efficient connection type due to a bug we have yet to identify. We hope that once we complete the software update to Mastodon this bug should be resolved.
2nd January 2024
No incidents reported
1st January 2024
No incidents reported
31st December 2023
No incidents reported
30th December 2023
No incidents reported
29th December 2023
No incidents reported
28th December 2023
No incidents reported
27th December 2023
No incidents reported
26th December 2023
No incidents reported
25th December 2023
No incidents reported
24th December 2023
No incidents reported
23rd December 2023
MastodonApp.UK Full Text Search Issues
The Elasticsearch instance currently running for MastodonApp.UK is experiencing some sort of issue which is preventing search operations running. Support are investigating the issue.
We have deployed new infrastructure and re-build our search system from the ground up. Everything should now be working as expected.
We have not progressed the work to restore the full text search capability as of yet, we will re-build the required infrastructure shortly.
