Assets failing to load in content Hub

Incident Report for Amplience

Postmortem

Issue summary

On the afternoon of Tuesday 25th July, we identified an issue with loading assets in Content Hub.

Root Causes

We identified shard failures which contributed to the loss of asset visibility from the Content Hub Interface. This also affected uploading new assets and updating due to the loss of visibility into existing and new assets.‌

Corrective actions

Internally, we have conducted discussions on how to enhance our ability to mitigate such a failure from occurring again. New research and development will be conducted around existing systems as well as looking at new methods to avoid disruption to live and existing content within the platform.

Posted Jul 31, 2023 - 15:40 BST

Resolved

We have rolled out a change to mitigate the failed shards, we are now seeing stability across Content Hub and will continue to monitor this evening.

Posted Jul 25, 2023 - 18:19 BST

Monitoring

We identified a second failure which will directly affect Assets loading in Content Hub. We will continue to monitor as shards recover.

Posted Jul 25, 2023 - 15:41 BST

Identified

We have identified a second failure which will directly affect Assets loading in Content Hub. We will continue to work on a secondary fix.

Posted Jul 25, 2023 - 15:35 BST

Monitoring

We have identified the issue and a fix applied. We can see assets are becoming available again and we will continue to monitor the situation.

Posted Jul 25, 2023 - 15:19 BST

Update

We are currently investigating a failed shard affecting assets loading on Content Hub

Posted Jul 25, 2023 - 15:01 BST

Investigating

We are currently investigating a failed shard affecting assets loading on Content Hub

Posted Jul 25, 2023 - 15:01 BST

This incident affected: Content Hub.