ATLAS Media Group Status Page

Some systems are experiencing issues.

ATLAS Media Group Status Page

The latest status updates from the team. Updates provided on known and ongoing incidents.

Past incidents

No incidents reported.

Affected Components: MastodonApp.UK Website

Fixed

7 months ago

Service has been restored.

Investigating

7 months ago

Due to other factors we haven't been able to fully investigate the issue as of yet. We will be starting to look into this shortly.

We are seeing increased error rates on the site as well as exceptionally poor performance while we use our older nodes, we are working to resolve the situation.

Reported

7 months ago

We are investigating a fault with our new web server that was built yesterday evening which appears to be partially breaking when viewing on the web.

For the time being we have failed back to our legacy infrastructure though this is causing considerable performance issues due to the latency between the web nodes and the database infrastructure. We are working to identify the fault on the new server and remedy the issue.

Fixed

7 months ago

We have now restored full service.

Reported

7 months ago

We're running into a series of issues with our load balancer infrasstructure which we think is related to both changes made yesterday evening as well as today. The primary LB is no longer starting as expected and due to experimental testing being undetrtaken to troubleshoot issues the secondary LB is currently pointing to our slower servers in some cases.

Multiple services are partially or fully degraded at this time.

MastodonAppUK - Database Migration

8 months ago
Complete
Affected Components: MastodonAppUK Database Service, MastodonApp.UK Website and MastodonAppUK Queue & Content Processing Service

We will be performing maintenance to switch our Database server from the legacy server cluster to our new server cluster. As part of this we will need to take the site offline for a short amount of time to facilitate the switch over to our new server infrastructure.

We expect the outage itself to last no more than approx 45 mins and will do what we can to minimise the disruption.

No incidents reported.

No incidents reported.

MastodonAppUK - Redis Migration

8 months ago
Complete
Affected Components: MastodonApp.UK Website, MastodonAppUK Advanced / Full Text Search, MastodonAppUK E-Mail Service and MastodonAppUK Queue & Content Processing Service
8 months ago

We have completed the Redis migration and all services are now fully restored and content processing queues are fully caught up.

Affected Components: Universeodon Queue & Content Processing Service

Fixed

8 months ago

Queues have now fully cleared and we're operating in real time with the wider network of servers.

Watching

8 months ago

Our ingest pipeline is down to running around 8 hours behind live at this time. All other queues appear to be operating as expected.

We are seeing a high error rate when interacting with our object storage endpoint especially when generating preview images for links so a large number of jobs are likely going to need to be manually re-ran.

Watching

8 months ago

We have restarted our content processing services which appear to have hung without any errors being recorded. We will continue to monitor the queues while they reduce. As of this update all queues with the exception of ingress are now running in real time and ingress is tracking approx 13 hours behind.

Reported

8 months ago

We are investigating reports of the queue service and content processing service being unresponsive.