Issue Summary:
On July 21st, 2026, the CyberGrants application experienced a service disruption between
approximately 10:10 AM ET and 12:00 PM ET. During this period, some users may have
experienced errors or been unable to access certain application features. The issue was
resolved following a database restart. No customer data was lost or corrupted.
Root Cause:
CyberGrants uses Oracle as its database platform. A bug within Oracle's database engine (a
component owned and maintained by Oracle) triggered a flood of internal errors that
overwhelmed the database's diagnostic subsystem, ultimately causing the database to
crash. On attempting to restart, residual memory corruption from the crash required an
additional restart cycle to clear. This is a confirmed Oracle product defect, not an issue with
the CyberGrants application code or Bonterra infrastructure. No system changes were made
by Bonterra prior to the failure occurring.
Prevention:
- Immediate: We restarted the database to clear the failed state and verified all data
remained intact. Downstream data replication processes were monitored through recovery
and were confirmed to be operationally correct.
- Short-term: We have opened a high-severity support case with Oracle to obtain a
permanent software patch that addresses this defect.
- Long-term: We are implementing enhanced monitoring to detect early warning signs of this
type of failure (abnormal internal error rates and diagnostic file growth), enabling our team to
intervene before users are significantly impacted. We are also working with Oracle to
identify configuration changes or patches that will prevent this defect from being triggered in
the first place.
- n.b.: This is the second issue in the space of 8 days which has been caused by Oracle
Database bugs. As a result, and in addition to the measures above, we are reviewing the
architecture of the Oracle DB within AWS and exploring an option to fundamentally change it
and thus making it more robust.