Past Incidents

Monitoring confirms full service is restored and stable.

We've applied patches to the host as part of a reboot of the VM and have re-started the Bungee software. Service appears to be operational and we're monitoring the situation.

We believe the root cause of the issue is down to the Bungee Server running out of memory, it's our current thought that there is a memory leak in the version of Bungee we're running. An upgrade is due for the bungee server which we hope to resolve the issue, in the mean time we've initiated a reboot of the bungee service which should restore service.

Service is fully operational.

A fix has now been rolled out to move the intensive read/write operations from the server running our primary database server onto some of our other infrastructure. We're confident this should resolve the poor performance that has been observed and will be monitoring over the coming 24 hours.

Apologies for the late update on the last comment, we incorrectly posted it on a previous incident. We have one final attempt we will make tonight to restore full performance to the environment and should this not work we will look to roll back the production impacting changes and get some sleep.

We are actively trying to push out a new infrastructure component to take the load off our primary database node, however data replication has been failing and we're working to identify the root cause of this new issue.

We are currently working to deploy a database replica on a new physical host which will allow us to move the cause of the heavy IO Usage onto that node and away from our current primary database node. Once complete we should start to see the site stabilize.