The ICS began issuing exception errors on all requests and creates from approximately 12:05 AEDT on Wednesday 4 October 2017. This affected the processing of all declarations, report types, underbond movements and status updates. Cargo Interactive users were unable to create or search for any work items.
The problem was identified as the database (DB2) internal Relative Byte Address (RBA) limitation. This caused DB2 to stop working. At approximately 02:00 on Thursday 5 October 2017 the ICS database was again able to process data existing in the system. New data did not start processing until around 07:00.
There was an increase in the number of calls to the Cargo Support Centre (CSC) from the start of the outage with over 1,000 calls received. The CSC also processed 5 requests for Urgent Clearance. Business Continuity processes were invoked from 14:30 until 08:30.
Investigations identified that DB2 uses an internal Relative Byte Address (RBA) or pointer to store all of its log entries. The RBA is a 6 byte field and continually updates the DB2 subsystem. Six months ago, the RBA was only 75% full – note that this DB2 subsystem was first started at the inception of ICS in 2004. The RBA reached its maximum limit and DB2 stopped processing. The overall impact was that the ICS in effect stopped processing any requests.
Industry was notified of the incident and provided with updates throughout the period via SMS, subscription emails, ICS announcements and ICS updates and through calls to the Cargo Support Centre.
Further information can be obtained by emailing firstname.lastname@example.org.