Dynamic Imaging Performance Issues
Incident Report for Amplience
Postmortem

Issue Summary:

During the morning of Friday, March 22nd, 0.1% of Dynamic Imaging resulted in 404 errors.

Initially, we attributed this issue to a potential incident with Cloudflare. https://www.cloudflarestatus.com/incidents/x1232bd91lkp

Amplience boasts a multi-layer CDN infrastructure, ensuring continuous delivery. We were able to redirect our traffic away from Cloudflare within 15 minutes while we investigated further.

Upon further analysis, we discovered that it stemmed from a recent change pushed out by Cloudflare to all environments. 

 

Root Causes:

Cloudflare, a third-party supplier, implemented a modification to HTTP/2, which took effect in our environment by default at 00:00. Disabling this modification resolved the service disruption. 

Corrective Actions:

Upon being informed by Cloudflare about the change, we proceeded with an internal risk assessment. This Incident has highlighted areas where further improvements are warranted.

Posted Mar 25, 2024 - 16:36 GMT

Resolved
The issue is now resolved. We will conduct an internal review and post our findings in a postmortem
Posted Mar 22, 2024 - 09:44 GMT
Monitoring
A third-party supplier experienced an outage which impacted some of our services.
They have implemented a fix and we are monitoring our services.
https://www.cloudflarestatus.com/incidents/x1232bd91lkp
Posted Mar 22, 2024 - 09:26 GMT
Investigating
We are currently investigating reports of broken images
Posted Mar 22, 2024 - 09:12 GMT
This incident affected: Dynamic Media.