Issue Summary: On 7th March 2025, at approximately 09:19 EST, we became aware that some Apricot clients may experience a downgraded performance when loading certain reports in the system. After investigation and rectification, the issue was resolved by 12:20 EST and normal performance resumed.
Root Cause: One element of the reporting infrastructure was experiencing 100% utilization under sustained load. The affected component was scaled up to manage this and additional workloads that could reasonably be anticipated in the future. Once this had been done, levels of utilization dropped back down to normal with a sensible buffer for spikes in demand.
Prevention: Additional monitoring and alerting implemented to catch potential spikes much earlier and automatically invoke a process for preventative rectification and scaling. The on-call runbook was appropriately updated as well.