Real time service status from across the ATLAS Media Group portfolio
No incidents reported
No incidents reported
We are continuing to see intermittent and significant outages on Universeodon. We're currently trying to pinpoint the root cause, however this is proving challenging.
We believe the significant increase in scale combined with updating settings on the Cloudflare CDN for Universeodon has helped to remove bot / spam traffic as well as increase capacity for our genuine users. We will be continuing to monitor over the longer term.
We have continued working overnight to try to come to a resolution for this issue, however have been unable to do so despite significant scaling and testing. Further investigation has to unfortunately be paused until this coming Saturday at which point we will re-investigate. The site has been primarily offline during this time and is likely to continue to be primarily offline during peak times.
Due to distance between the web tier that users are accessing and the database and other components behind the scenes there is currently a significantly increased latency across the Universeodon website. We are working to migrate the web tier onto new infrastructure which should be on a more local and lower latency network.
Due to database capacity issues as a result of an earlier failure in one of our underlying components, there are intermittent outages due to the database being unable to take the extra load. We are working to mitigate this issue.
We have been able to remove the majority of the load from the database however we are seeing other issues causing intermittent outages.
We believe the intermittent outages should now be resolved after moving the web tier to a server more geographically practical with an improved connection. We are still seeing an increase in errors however believe this is related to the latency issues which are yet to be resolved.
We are seeing major delays to content processing on Unvierseodon.com - We are working to scale up additional capacity at this time to help process the queues quickly.
Queues have returned pretty much to normal now. We will continue monitoring and a migration is planned where we will be able to better and more efficiently scale things.
One of the components that helps manage load to our database server has failed and as a result connections are starting to overwhelm the database itself. We're continuing to monitor the situation and are seeing improvements with the ingress queue however the internal queues and other queues are degrading further at this time.
No incidents reported
No incidents reported
No incidents reported
No incidents reported
No incidents reported
No incidents reported
No incidents reported
We are investigating a major outage of the MastodonApp.UK site - We believe this is due to underlying network changes that the team is working on. Updates to follow.
Retroactive Close
The restart of the edge router appears to have restored full service to the applications. We will continue to monitor.
We have reverted our configuration changes however are not yet seeing improvement in service. We are restarting our edge router for the host involved now and will continue to work towards a resolution.
We have identified that the issue is the result of a loss of internet connectivity to our physical server running the database. KbinApp.UK is also now impacted.
We have identified an issue with our search service after the reboot of the physical host it sits on earlier today which we believe is impacting views for the advanced dashboard and may be impacting other aspects of the service.
There were two issues, the first being a false positive after my local Firefox updated, and it turned out AdBlocker Ultimate was blocking a number of our cached files. I would suggest (given we don't actually serve any adds!) that you add an exception for mastodonapp.uk and it's subdomains if you have an add-blocker.
Our second issue was the Elasticsearch container running on one of our VM's did not auto start, despite the container auto starting. We are looking to fix this for the long term!
We are applying updates to the underlying host for MastodonApp / KbinApp which will impact MastodonApp Content processing, Text Search and KbinApps Redis (and therefore impacting the entire web application).