Summary
On Wednesday, February 8th, we experienced a disruption that impacted the availability of our Search index. This issue affected Dynamic Content users searching for content or linking new content.

What happened
We carried out a regular release in our production environments to upgrade users' ability to search for delivery keys within their content items. During this process, we encountered an issue where the disk space allocated to the service was not scaling at the rate expected and was rapidly approaching its capacity limit. The shortage of disk space caused the service to become unstable and prevented it from writing to the search index.
Our Engineers resolved the issue by manually adding additional capacity to the service. During this period, we also had to halt the search index process and allocate more time for the process to complete.

Our Course of Action Regarding This Issue
After the recent disruption, we thoroughly evaluated the cause of the issue and determined a series of events that led to the failure. Our engineering team has taken swift action to address these findings and put measures in place to prevent similar incidents from happening in the future. Some of the steps we have taken to correct the issue include:-

• Any planned updates to our search index will be communicated in advance.
• We have implemented additional alerting to capture events around low disk space or failed writes.
• We have documented the steps taken to resolve the recent issue with disk space and will use this information to improve our processes and prevent similar incidents from reoccurring.

‌

We apologize for any inconvenience this may have caused and are taking steps to prevent similar issues from occurring in the future. If you have any questions, please reach out to support@amplience.com.