ATLAS Media Group Status Page

Some systems are experiencing issues.

ATLAS Media Group Status Page

The latest status updates from the team. Updates provided on known and ongoing incidents.

Planned maintenance

  • We need to perform routine maintenance to our physical hardware located within the Redditch Data center. As part of this we will need to take all services offline for a short amount of time while we apply system and security updates to our shared storage pool and restart the underlying host. We will also be performing patching and upgrades to our application hosts during this time and will be failing over our applications across the nodes in the environment.

    We expect the maintenance itself to take approx 3 hours with a short outage of up to 30 mins throughout this while we take the environment offline for full system restarts and hardware improvements.

    Affected Components: VPS Control Panel, Universeodon Database, Universeodon Queue & Content Processing Service, Universeodon Website, Universeodon Relay, Billing Panel, Blog, MastodonApp.UK Website, MastodonAppUK Database Service and MastodonAppUK Queue & Content Processing Service

Past incidents

No incidents reported.

No incidents reported.

No incidents reported.

No incidents reported.

No incidents reported.

Fixed

4 months ago

Full service has been restored.

Reported

5 months ago

We're seeing intermittent outages on MastodonAppUK and are investigating the issue.

5 months ago
Fixed

Fixed

5 months ago

Full service has been restored.

Watching

5 months ago

Our backups are now in a better state, we're re-starting the Universeodon service at this time.

Identified

5 months ago

We're continuing to see a large backlog in WAL Files needing to be pushed to our external backup service. We're going to continue to monitor and we will look to keep the site offline until we've made a significant enough dent in the backlog to safely restore service and not risk backup stability.

Identified

5 months ago

Due to growing disk usage and our backups struggling to keep up right now we've temporarily taken Universeodon.com offline to allow the backups a chance to fully catch-up. Unfortunately once we increase the disk allocation to the DB Server it's impossible to reverse and with the current constraints with SSD Storage globally we are looking to conserve capacity where possible.

Identified

5 months ago

We can see a large backlog in WAL Files waiting to be pushed to our off-site backup service, this is likely the result of the issues with our networking and routing which has taken DNS offline a few times. We expect that once all these files are pushed up we should see a substantial amount of disk space released to the DB Server.

Investigating

5 months ago

Despite a significant increase in disk capacity we're once again seeing disk related issues impacting Universeodon - We suspect this might be the result of issues with our backup streaming. We are investigating.

Watching

5 months ago

We have expanded the disk space and will be reviewing our monitoring tool configuration to ensure high disk capacity warnings are flagged in future to allow us to proactively intervene. We will continue to monitor to ensure full stability has restored. As part of this we are also validating our off-site backup configuration has not been impacted and is still operational.

Identified

5 months ago

We have identified the issue as being a full disk on our primary database server. We're working to expand capacity now and restore service as quickly as possible.

Reported

5 months ago

Monitoring has detected a global outage of Universeodon - We are investigating.