Some systems are experiencing issues.
We need to perform routine maintenance to our physical hardware located within the Redditch Data center. As part of this we will need to take all services offline for a short amount of time while we apply system and security updates to our shared storage pool and restart the underlying host. We will also be performing patching and upgrades to our application hosts during this time and will be failing over our applications across the nodes in the environment.
We will additionally be replacing the core networking equiptment located within our rack as part of a major overhaul and long term improvement plan to our capabilities at our Redditch location. We expect this to result in an extended outage while we troubleshoot any unexpected issues and re-wire the networking infrastructure on-site. While certain aspects of the hardware change will be able to be done prior to us needing to take the environment offline a significant portion of the work will result in disruption.
We expect the maintenance itself to take approx 5 hours with a short outage of up to 30 mins for patching and associated reboots and a further 60 min outage during this time to replace the core networking infrastructure.
We have now reached the point where all services except the SN VPSCP are fully operational. We're going to raise a new incident for the VPSCP which will be investigated later on today.
We have now completed the restoration of service to some of our direct downstream customers. We are now moving on to restore the outstanding services including the Universeodon Relay, FediMedi, The Superior Networks VPSCP and Billing panel and other associated services.
MastodonAppUK is now back operational, we do have reduced capacity for the site currently both with regards to the website itself and our back-end content processing. Universeodon content processing is coming back online and the website is at it's standard capacity (Which is unfortunately always reduced at the moment).
We are continuing to work to stabalize the infrastructure after these major upgrades and are working to get everything operational as quickly as possible.
We have managed to partially restore the MastodonAppUK Service, we're currently working to now get Universeodon back online and will resume work for one of our downstream customers who is also experiencing network issues as a result of this change.
We are continuing to work to restore customer connectivity at our Redditch location and will have a further update shortly.
We continue to see major issues with our new networking environment. We are currently prioritising bringing our dedicated server and colo customers that have been impacted back online after which we will resume restoration of our other service offerings.
We have now managed to restore the internal network to the site with the upgraded hardware. We're working now to get public access to our load balancer infrastructure restored before we move on to restoring services. Apologies again for the extended delay in this process.
We have continued to experience issues with our new networking setup. Our team are working hard to resolve the issue as quickly as possible.
We are now starting the next phase of work which will involve a loss of networking for a short period of time while we replace some of our core networking infrastructure. We will provide updates as we progress.
We have completed the first stage of the maintenance updating the underlying storage systems firmware and software to the latest release. We're confirming that all services are operational again at this time before we commence the next phase of works.
We are going to commence maintenance at this time. Further updates to follow.