Past Incidents

We are confident that full service is restored. For the time being we are not reactivating the CDN for MastodonAppUK and Universeodon.com as initial feedback suggests the performance is better without the CDN. Media and the relay will continue to route via the CDN.

Monitoring detects that all services have been restored and Cloudflare report the majority of their systems are now operational. We've had to re-enable the Cloudflare proxy in front of the relay as it looks like some clients are not recognising the LetsEncrypt certificate being used however for the time being we're going to leave the MastodonAppUK site and Universeodon site directly connecting to our infrastructure to ensure all services fully recover.

For MastodonAppUK, Universeodon.com and the Universeodon relay we have disabled routing via the Cloudflare CDN for the time being. This should mean our sites will be online and operational though may be with increased latency for users further from the UK. We are continuing to monitor the situation.

We are unable to do a lot to mitigate these issues, if absolutely necessary we may be able to re-route our traffic away from the CDN and directly to our DC routers however this is likely to have a major performance impact for users further away from our UK based servers. We will continue to monitor the situation while Cloudflare work to remediate. You can follow the cloudflare incident here - https://www.cloudflarestatus.com/incidents/8gmgl950y3h7

We can see the issue is with Cloudflares global network and appears to be intermittent, our monitoring continues to confirm there are global outages intermittently across all of our sites which use Cloudflare as a CDN.

The issue appears to be far more widespread with issues now impacting the Universeodon relay and the Superior Networks website as well. We're also seeing outages across our media endpoints and other services. We believe this to be an issue with Cloudflare.

All services now restored.

We're running into a lot of issues getting our storage server to re-connect to the network as well as our second app server, until these are fully online we won't be able to get service restored. We're working as fast as we can to bring everything back online. Apologies for the delay.

The changes should now be complete, we're working to get the servers re-connected and operational as the second server is having some issues with network connectivity. Once this is complete the site should be restored. We're working as fast as we can to get everything operational.

Getting the new gateway operational is taking a bit longer than we expected. We're now hoping to get everything restored by 15:00 UTC. Apologies for the delay.

The upgrades to our storage server have completed and all is operational there now. We're monitoring it to ensure it remains stable and will shortly be powering down our gateway to replace it with the new one.

We have taken the time to also complete the cable adjustments while the server was upgrading so we hope that once the gateway switch is complete and everything is back up and running there that we should have all systems operational with relative ease.

We're currently performing a software update to our existing gateway device to ensure the new one and the old one can run the latest firmware and that there are no issues with the move.

Maintenance is now under way and we've shut down our application servers.

The first step now for us is to apply the relevant updates to our underlying storage server, we can't do this while the applications are running easily without risking issues with the VM's access to the underlying storage so we're working through this now. Once that is complete we need to swap out the gateway device which has been causing us significant issues over the last couple of months.

We will later also need to make some cabling changes however this should all be on our APP-2 server and we should be able to get the site operational before we make these changes, it will impact our ability to run content processing but it shouldn't cause further site outages.