The ICS began experiencing processing delays from approximately 05:30 on Wednesday 14 June 2017. The processing delays became significant as the load on the system increased across the business day. The delays affected the processing of all report types, underbond movements and status updates. There were intermittent errors for Cargo Interactive users when trying to access some screens. The ICS processing caught up towards the end of the day with all transactions processed and queues cleared by 21.00.
The problem was identified as poor database access path issues that unexpectedly occurred after the introduction of new tables from ICS release 17.4.02. The changes had been tested successfully in the lower environments prior to going into Production. Technical remediation was applied throughout the day as poor performing transactions were identified. To alleviate the impact on Industry, air cargo screening periods were reduced to 1 hour from 12:19 until 17:00. Furthermore, the Department purchased extra processing capacity to assist with processing the backlog.
There was an increase in the number of calls to the Cargo Support Centre (CSC) with over 1,110 calls received. The CSC also processed 7 requests for Urgent Clearance. Business Continuity processes were invoked from 09:45 until 16:00. During this period the CSC received 183 BCP files and processed 62 successfully.
Investigations identified that as part of ICS release 17.4.02 a number of DB2 tables were changed affecting a range of transactions. The “bind” process should have automatically selected the most effective access paths for these changed tables. This did not occur. The resulting inefficient database access paths caused a number of high volume processes running for extend periods, locking and consuming a large amount of system resources. The overall impact was a major degradation in system performance.
Industry was notified of the incident and provided with updates throughout the day via SMS, subscription emails, ICS announcements and ICS updates and through calls to the Cargo Support Centre.
Further information can be obtained by emailing firstname.lastname@example.org.