Staging links return 504 errors

Incident Report for Amplience

Postmortem

Incident Start Time: 15.56 GMT 

Incident End Time: 16.20 GMT 

Issue summary

On Tuesday, 4th March, users of Virtual Staging Environments (VSE) experienced 504 Gateway Timeouts errors when attempting to preview content and assets. The delivery of published content and assets were not impacted. 

Root cause

Earlier in the day, we deployed a low-risk update to the VSE stack, which was successfully validated post-deployment. However, the service was not scaled appropriately to accommodate the traffic demands, ultimately reaching capacity and triggering 504 errors. 

The issue was resolved by scaling the service appropriately.

Corrective Actions 

Future releases to the VSE will include proactively monitoring the drift within Cloudformation to ensure service capacity meets traffic requirements.  

‌If you would like to discuss this further, please contact your customer success manager (csteam@amplience.com)

Posted Mar 07, 2025 - 14:06 GMT

Resolved

This incident is now resolved.
Posted Mar 04, 2025 - 16:22 GMT

Monitoring

A fix has been implemented, and we are monitoring the results.
Posted Mar 04, 2025 - 16:20 GMT

Investigating

We are currently investigating an issue where staging (Virtual Staging Environements) links return 504 error codes. Existing published images are not impacted.
Posted Mar 04, 2025 - 16:16 GMT
This incident affected: Content Hub and Dynamic Media.