Network outage Friday 7th November 2025 00:25:00

The gateway has finally become responsive again, it looks like the CPU and load averages were maxed out causing the processes to crash, we're monitoring the gateway at this time and have restored access to both sites and re-enabled content processing.

I've managed to get some access into the network despite the router itself being pretty much entirely unresponsive. I am hoping to remotely reboot the device as soon as I can gain access via the cluster which will hopefully start to restore functionality.

We have once again lost network access to our gateway device. We are working to see if there is anything we're able to do to restore access at this time.

We've had a lot of issues throughout the day with connectivity between our new DC site and our legacy sites which has resulted in our content processing struggling to process in a timely manner. In the case of MastodonAppUK we're now mostly back on track though had earlier issues where almost all jobs failed and needed to be re-tried which skewed our data. Universeodon.com still has around 75k ingest jobs to run which are slowly being processed but again are having connectivity related issues due to issues with our network hardware and connectivity issues with our legacy environment.

Our network gateway has been restarted by the on-site team. We did run into some issues when the site came back online due to the volume of traffic being generated through the content federation and federation updates, we've scaled back some of the content processing for the time being to try to keep the site running stable.

We have agreed with the hardware vendor to return this model of hardware and will be ordering and replacing it with a more powerful version which we hope will operate with greater stability.