Bulk upload failure

Incident Report for Amplience

Postmortem

Incident Start Time: 19/01/2025 22.00 GMT

Incident End Time: 21/01/2025 12.30 GMT

‌

Issue summary

On Sunday night, some bulk ingestion jobs were initiated but did not complete, resulting in them becoming stuck.

‌

Root cause

This was due to unexpected behaviour from routine repair services on our back-end database.

‌

Corrective actions

Additional alarms have been implemented to monitor and capture this new behaviour.

Upgrades to this part of our platform are already underway to improve stability and performance.

‌

‌If you would like to discuss this further, please contact your customer success manager (csteam@amplience.com)

Posted Jan 23, 2025 - 11:11 GMT

Resolved

This incident has been resolved.

Posted Jan 21, 2025 - 14:41 GMT

Monitoring

A fix has been implemented and we are monitoring the results.

Posted Jan 21, 2025 - 14:32 GMT

Identified

Our operations team have confirmed the backlog of stuck processing jobs have now cleared.

Posted Jan 21, 2025 - 12:36 GMT

Investigating

We are currently investigating the cause of the bulk upload issues experienced over the last 24 hours, we will provide further updates as soon as possible.

Posted Jan 21, 2025 - 10:40 GMT

This incident affected: Content Hub.