Past Incidents

We are confident that full service is restored. For the time being we are not reactivating the CDN for MastodonAppUK and Universeodon.com as initial feedback suggests the performance is better without the CDN. Media and the relay will continue to route via the CDN.

Monitoring detects that all services have been restored and Cloudflare report the majority of their systems are now operational. We've had to re-enable the Cloudflare proxy in front of the relay as it looks like some clients are not recognising the LetsEncrypt certificate being used however for the time being we're going to leave the MastodonAppUK site and Universeodon site directly connecting to our infrastructure to ensure all services fully recover.

For MastodonAppUK, Universeodon.com and the Universeodon relay we have disabled routing via the Cloudflare CDN for the time being. This should mean our sites will be online and operational though may be with increased latency for users further from the UK. We are continuing to monitor the situation.

We are unable to do a lot to mitigate these issues, if absolutely necessary we may be able to re-route our traffic away from the CDN and directly to our DC routers however this is likely to have a major performance impact for users further away from our UK based servers. We will continue to monitor the situation while Cloudflare work to remediate. You can follow the cloudflare incident here - https://www.cloudflarestatus.com/incidents/8gmgl950y3h7

We can see the issue is with Cloudflares global network and appears to be intermittent, our monitoring continues to confirm there are global outages intermittently across all of our sites which use Cloudflare as a CDN.

The issue appears to be far more widespread with issues now impacting the Universeodon relay and the Superior Networks website as well. We're also seeing outages across our media endpoints and other services. We believe this to be an issue with Cloudflare.

All services now restored.

We're running into a lot of issues getting our storage server to re-connect to the network as well as our second app server, until these are fully online we won't be able to get service restored. We're working as fast as we can to bring everything back online. Apologies for the delay.

The changes should now be complete, we're working to get the servers re-connected and operational as the second server is having some issues with network connectivity. Once this is complete the site should be restored. We're working as fast as we can to get everything operational.

Getting the new gateway operational is taking a bit longer than we expected. We're now hoping to get everything restored by 15:00 UTC. Apologies for the delay.

The upgrades to our storage server have completed and all is operational there now. We're monitoring it to ensure it remains stable and will shortly be powering down our gateway to replace it with the new one.

We have taken the time to also complete the cable adjustments while the server was upgrading so we hope that once the gateway switch is complete and everything is back up and running there that we should have all systems operational with relative ease.

We're currently performing a software update to our existing gateway device to ensure the new one and the old one can run the latest firmware and that there are no issues with the move.

Maintenance is now under way and we've shut down our application servers.

The first step now for us is to apply the relevant updates to our underlying storage server, we can't do this while the applications are running easily without risking issues with the VM's access to the underlying storage so we're working through this now. Once that is complete we need to swap out the gateway device which has been causing us significant issues over the last couple of months.

We will later also need to make some cabling changes however this should all be on our APP-2 server and we should be able to get the site operational before we make these changes, it will impact our ability to run content processing but it shouldn't cause further site outages.

Service health was restored.

The gateway has finally become responsive again, it looks like the CPU and load averages were maxed out causing the processes to crash, we're monitoring the gateway at this time and have restored access to both sites and re-enabled content processing.

I've managed to get some access into the network despite the router itself being pretty much entirely unresponsive. I am hoping to remotely reboot the device as soon as I can gain access via the cluster which will hopefully start to restore functionality.

We have once again lost network access to our gateway device. We are working to see if there is anything we're able to do to restore access at this time.

We've had a lot of issues throughout the day with connectivity between our new DC site and our legacy sites which has resulted in our content processing struggling to process in a timely manner. In the case of MastodonAppUK we're now mostly back on track though had earlier issues where almost all jobs failed and needed to be re-tried which skewed our data. Universeodon.com still has around 75k ingest jobs to run which are slowly being processed but again are having connectivity related issues due to issues with our network hardware and connectivity issues with our legacy environment.

Our network gateway has been restarted by the on-site team. We did run into some issues when the site came back online due to the volume of traffic being generated through the content federation and federation updates, we've scaled back some of the content processing for the time being to try to keep the site running stable.

We have agreed with the hardware vendor to return this model of hardware and will be ordering and replacing it with a more powerful version which we hope will operate with greater stability.