We are seeing major delays to content processing on Unvierseodon.com - We are working to scale up additional capacity at this time to help process the queues quickly.
Queues have returned pretty much to normal now. We will continue monitoring and a migration is planned where we will be able to better and more efficiently scale things.
One of the components that helps manage load to our database server has failed and as a result connections are starting to overwhelm the database itself. We're continuing to monitor the situation and are seeing improvements with the ingress queue however the internal queues and other queues are degrading further at this time.
12th August 2023
No incidents reported
11th August 2023
No incidents reported
10th August 2023
No incidents reported
9th August 2023
No incidents reported
8th August 2023
No incidents reported
7th August 2023
No incidents reported
6th August 2023
No incidents reported
5th August 2023
MastodonApp.UK WebsiteMastodonApp.UK Outage
We are investigating a major outage of the MastodonApp.UK site - We believe this is due to underlying network changes that the team is working on. Updates to follow.
Retroactive Close
The restart of the edge router appears to have restored full service to the applications. We will continue to monitor.
We have reverted our configuration changes however are not yet seeing improvement in service. We are restarting our edge router for the host involved now and will continue to work towards a resolution.
We have identified that the issue is the result of a loss of internet connectivity to our physical server running the database. KbinApp.UK is also now impacted.
Advanced SearchSearch Outage - Impacting hashtag / similar searching
We have identified an issue with our search service after the reboot of the physical host it sits on earlier today which we believe is impacting views for the advanced dashboard and may be impacting other aspects of the service.
There were two issues, the first being a false positive after my local Firefox updated, and it turned out AdBlocker Ultimate was blocking a number of our cached files. I would suggest (given we don't actually serve any adds!) that you add an exception for mastodonapp.uk and it's subdomains if you have an add-blocker.
Our second issue was the Elasticsearch container running on one of our VM's did not auto start, despite the container auto starting. We are looking to fix this for the long term!
Queue Service[MAINTENANCE] Updates to Prod-Lon-Dedi-01 - Impacting MastodonApp and KbinApp
We are applying updates to the underlying host for MastodonApp / KbinApp which will impact MastodonApp Content processing, Text Search and KbinApps Redis (and therefore impacting the entire web application).
