Bulk upload failure
Incident Report for Amplience
Postmortem

Incident Start Time: 19/01/2025 22.00 GMT

Incident End Time: 21/01/2025 12.30 GMT

Issue summary

On Sunday night, some bulk ingestion jobs were initiated but did not complete, resulting in them becoming stuck.

Root cause

This was due to unexpected behaviour from routine repair services on our back-end database.

Corrective actions

Additional alarms have been implemented to monitor and capture this new behaviour.

Upgrades to this part of our platform are already underway to improve stability and performance.

‌If you would like to discuss this further, please contact your customer success manager (csteam@amplience.com)

Posted Jan 23, 2025 - 11:11 GMT

Resolved
This incident has been resolved.
Posted Jan 21, 2025 - 14:41 GMT
Monitoring
A fix has been implemented and we are monitoring the results.
Posted Jan 21, 2025 - 14:32 GMT
Identified
Our operations team have confirmed the backlog of stuck processing jobs have now cleared.
Posted Jan 21, 2025 - 12:36 GMT
Investigating
We are currently investigating the cause of the bulk upload issues experienced over the last 24 hours, we will provide further updates as soon as possible.
Posted Jan 21, 2025 - 10:40 GMT
This incident affected: Content Hub.