Redditch DC Network Device Outage Thursday 2nd October 2025 02:00:00


At approx 2AM this morning our gateway device which provides internet access to all of our services stopped connecting back to the internet. We are waiting now for on-site techs to reboot the device. We are unsure as to what caused this as it's a freshly replaced device.

As of last night we have identified and resolved the fault that was causing instability on our network device as well as performance issues. We're now seeing noticeably better performance on Universeodon and our content processing is also running notably faster.

We have been able to restart the network device once again and it so far appears to be running stable. We have moved our Universeodon Redis and SideKiq content processing onto a different host within the cluster which has some slightly different physical networking properties which we hope will make it easier to deal with any future disruptions until the physical networking for the second app server is updated. Universeodon is currently online though we have a lot of content that will need processing as servers continue to retry sending us their events and updates. Apologies for the disruption, we are now awaiting a diagnostics update from Ubiquti who is the vendor for our networking hardware to identify why this has happened.

The team at the DC have rebooted the device for us, unfortunately it went back offline fairly quickly. We are investigating what's causing this behaviour.