Unable to Publish or Schedule in Dynamic Content

Incident Report for Amplience

Postmortem

Incident Start Date: 19/08/2024 17:59 GMT
Incident End Date: 20/08/2024 01:50 GMT

Issue

During the incident, some users were unable to schedule content for publishing. The UI indicated that the publishing process was hanging, preventing users from successfully completing their tasks.

Root Cause

The issue was traced back to an undetected faulty shard within our infrastructure. Any publishing job routed to this shard was not processed, leading to the observed hang in the UI.

Corrective Actions

To prevent this from happening in the future, we have implemented the following measures:

Additional Monitoring: We have added new monitors that will alert us if any shard starts to fall behind in processing. This will enable us to detect and address similar issues more quickly.
Infrastructure Protocols: We are adding additional protocols to better protect our infrastructure and ensure a more robust and resilient system.

Posted Aug 21, 2024 - 15:27 BST

Resolved

The publishing status backlog is now back to the normal level and content items should now be publishing as expected.

Posted Aug 20, 2024 - 07:08 BST

Monitoring

We have identified that there was a backlog of publishing jobs due to a stuck event listener. It appears that the content is publishing but there is a delay in the published tick showing in Dynamic Content. We expect the publish tick to catch up over the next few hours.

Posted Aug 20, 2024 - 00:23 BST

Update

Posted Aug 20, 2024 - 00:21 BST

Investigating

Some Customers are experiencing issues with publishing and scheduling in Dynamic Content. We are currently working to identify the problem

Posted Aug 19, 2024 - 21:12 BST

This incident affected: Dynamic Content.