Virtual Staging Environment (VSE) Performance Issues
Incident Report for Amplience
Postmortem

Summary

On Monday, September 18th, we experienced 500 errors from images using the Amplience Virtual Staging Environment endpoint.

Root Cause

We discovered failures after a renewed certificate used on multiple services failed. One of the our internal services did not have an up to date root level certificate and was therefore unable to validate against the latest certificate change. This in turn caused the service to fail. Once we detected failures we begun the process of rolling back to the previous certificate and we are now in the process of working towards a permanent fix.

Corrective and Preventive Measures

Internally, we are working towards a permanent fix going forward.

Posted Sep 19, 2023 - 17:21 BST

Resolved
We have monitored traffic and error logs. We can confirm VSE is operational. We do apologise for any inconveniences this may have caused.
Posted Sep 18, 2023 - 20:14 BST
Monitoring
Our Engineering team have rolled back a change which you should now see vse working as expected
Posted Sep 18, 2023 - 19:52 BST
Identified
We have identified a failure with VSE. Our Operations and Engineering teams are currently investigating the cause and looking to fix the problem
Posted Sep 18, 2023 - 19:45 BST
This incident affected: Dynamic Content.