We are continuing to see intermittent and significant outages on Universeodon. We're currently trying to pinpoint the root cause, however this is proving challenging.
We believe the significant increase in scale combined with updating settings on the Cloudflare CDN for Universeodon has helped to remove bot / spam traffic as well as increase capacity for our genuine users. We will be continuing to monitor over the longer term.
We have continued working overnight to try to come to a resolution for this issue, however have been unable to do so despite significant scaling and testing. Further investigation has to unfortunately be paused until this coming Saturday at which point we will re-investigate. The site has been primarily offline during this time and is likely to continue to be primarily offline during peak times.
14th August 2023
KBinApp.UK WebsiteKbinApp.UK outage
We are aware of a major issue with the Kbinapp site. We have been looking into this issue for the last few days. This is a retrospective incident update.
The KbinApp Service has been retired.
WebsiteHigh Latency on Web and API for Universeodon
Due to distance between the web tier that users are accessing and the database and other components behind the scenes there is currently a significantly increased latency across the Universeodon website. We are working to migrate the web tier onto new infrastructure which should be on a more local and lower latency network.
Retroactive Close
The majority of the latency once the site has been loaded is now resolved. We still have further components to migrate which will (we expect) further speed up page loading.
WebsiteIntermittent Outages due to high load
Due to database capacity issues as a result of an earlier failure in one of our underlying components, there are intermittent outages due to the database being unable to take the extra load. We are working to mitigate this issue.
We have been able to remove the majority of the load from the database however we are seeing other issues causing intermittent outages.
We believe the intermittent outages should now be resolved after moving the web tier to a server more geographically practical with an improved connection. We are still seeing an increase in errors however believe this is related to the latency issues which are yet to be resolved.
13th August 2023
WebsiteUniverseodon Content Processing Delays
We are seeing major delays to content processing on Unvierseodon.com - We are working to scale up additional capacity at this time to help process the queues quickly.
Queues have returned pretty much to normal now. We will continue monitoring and a migration is planned where we will be able to better and more efficiently scale things.
One of the components that helps manage load to our database server has failed and as a result connections are starting to overwhelm the database itself. We're continuing to monitor the situation and are seeing improvements with the ingress queue however the internal queues and other queues are degrading further at this time.
