We are investigating a major outage of the MastodonApp.UK site - We believe this is due to underlying network changes that the team is working on. Updates to follow.
Retroactive Close
The restart of the edge router appears to have restored full service to the applications. We will continue to monitor.
We have reverted our configuration changes however are not yet seeing improvement in service. We are restarting our edge router for the host involved now and will continue to work towards a resolution.
We have identified that the issue is the result of a loss of internet connectivity to our physical server running the database. KbinApp.UK is also now impacted.
Advanced SearchSearch Outage - Impacting hashtag / similar searching
We have identified an issue with our search service after the reboot of the physical host it sits on earlier today which we believe is impacting views for the advanced dashboard and may be impacting other aspects of the service.
There were two issues, the first being a false positive after my local Firefox updated, and it turned out AdBlocker Ultimate was blocking a number of our cached files. I would suggest (given we don't actually serve any adds!) that you add an exception for mastodonapp.uk and it's subdomains if you have an add-blocker.
Our second issue was the Elasticsearch container running on one of our VM's did not auto start, despite the container auto starting. We are looking to fix this for the long term!
Queue Service[MAINTENANCE] Updates to Prod-Lon-Dedi-01 - Impacting MastodonApp and KbinApp
We are applying updates to the underlying host for MastodonApp / KbinApp which will impact MastodonApp Content processing, Text Search and KbinApps Redis (and therefore impacting the entire web application).
