Past Incidents

Both sites were fully online shortly after I posted this update, apologies for the slow status page update.

We are back fully online.

We're currently in the process of re-building all our search content, the vast majority of content should now be able to be searched over and search should mostly work.

We completed the deployment of a new physical host and have migrated our web servers onto this new host which appears to have resolved our intermittent outages.

MastodonApp.UK Content processing has now been migrated to a different server and is operational though working through a large backlog. An additional server is currently mid re-build and once complete will give us an additional web server for both sites which should both allow better performance in general but should also resolve some of the intermittent issues we've been seeing.

To try to reduce the impact of the issue we will be migrating the MastodonApp.UK Content processing worker onto a different host, the hope being this will free up capacity on the original host which is currently struggling with CPU capacity.

Storage has been recovered and is now operational. We will be migrating the outstanding workloads back to storage off of the shared pool we setup.

Universeodon should now be fully online. MastodonApp.UK continues to have issues due to the database being stopped without warning. We're looking into what's causing this issue.

Changes have been updated on our configuration and infrastructure. Queue processing time is now back to normal.