Our content processing queues appear to have partially crashed at around 8AM UTC today resulting in no new ingress of posts and content. Limited other actions will have been maintained, we're currently catching up on the processing backlog and expect disruption to continue for the next 1-2 hours.
We can confirm this issue was resolved last night, we've continued to monitor the situation overnight and this morning and are happy to confirm all queues are caught up.
We are continuing to see delays on content processing with around 80,000 items on our ingest queue however we currently lack the compute in our cluster to process these at a faster rate without causing major disruption to other services. We're tracking around 1 hour delay currently between events being entered into the queue and them being processed.
We are continuing to monitor the situation. We are currently tacking approx 2 hours behind on our ingest queues though all other queues at this time are showing clear.
