> It was discovered that there was a change made to the alarm text during this AGC update that caused the failure. Prior to the update, when the PII value exceeded +/-999 MW, the value in the alarm text defaulted to “xxxx,” preventing dispatchers from having an accurate indication of the amount of PII change. During this deployment, the two MW value fields in the PII alarm text were modified from i4 to i5 (4 digit integers to 5 digit integers) to allow for an additional digit.

> The original alarm text array had 79 characters which almost hit the max character limit of 80, and the change from i4 to i5 resulted in the alarm text increasing by 2 characters to 81 characters in length. Writing an 81 character text string into an 80 character fixed length array resulted in a run-time abort of the task.

I bet nobody expected that modifying a string can crash the entire control system due to a (caught) buffer overflow. Apparently the reason it went undetected was that the modification was not tested on a demo system first, since it was believed to be trivial.

> Lesson Learned: No matter how small the change is, validate changes in a test environment first. This requires using a thorough test script for the changes.