Content Delivery, Dynamic Content, Content Hub Performance Issues

Incident Report for Amplience

Postmortem

Incident Summary

Duration: 47 minutes (08/06/21 10.57 BST – 08/06/2021 11.44 BST)

On the morning of Tuesday 8th June 2021, an Amplience supplier; Fastly experienced a Global CDN Incident. This impacted the following Amplience Services:

• Content Delivery
• Content Delivery v2
• Dynamic Imaging

This also impacted delivery of content & images within Amplience platforms; Dynamic Content and Content Hub.

Amplience Engineers redirected traffic from Fastly to our Origin servers and scaled up our servers to absorb the traffic. Once we were confident that Fastly had resolved their Incident, we reverted traffic back to Fastly.

Root Cause

Fastly the Global CDN provider experienced a global node disruption caused by a change in configuration via one of their consumers

Corrective actions

• Review our auto failover rules so that we can redirect traffic quicker
• Review our delivery TTL’s when changes are applied
• Adjust our Fastly configuration to serve from stale

If you wish to discuss this further, please reach out to your Customer Success Manager (csteam@amplience.com)

Posted Jun 10, 2021 - 15:12 BST

Resolved

The incident is now resolved.
The Incident resolution time was 11.44 BST.

Posted Jun 08, 2021 - 13:45 BST

Monitoring

Fastly have finished rolling out their fix and we have finished re-routing traffic.
Customers should see services starting to resume

Posted Jun 08, 2021 - 12:00 BST

Update

At 11.44, Fastly confirmed that they have identified the issue and are currently fixing

Posted Jun 08, 2021 - 11:48 BST

Identified

We have identified that our provider; Fastly is having major network issues which is impacting a number of our services. We are working to mitigate this and will update you shortly.

Posted Jun 08, 2021 - 11:17 BST

Investigating

The issue has been identified, we are currently working to implement a fix.

Posted Jun 08, 2021 - 11:09 BST

This incident affected: Content Hub, Dynamic Content, and Dynamic Media.