The code isn't quite as terrible if compiled with optimizations on. I like your hypothesis elsewhere in the thread, that the optimization pass is probably removing unneeded array copying that hinder the performance of the debug build.
Semantically, an Array is an automatically resizing, reference-counted, copy-on-write, bounds-checking sequence of elements, none of which I want if I'm building a buffer for pixel values. So there is a significant mismatch between my desired use case and the characteristics of Array. I really do think there should be a fixed-size non-copy-on-write collection type at some point added to the language.
From an operational perspective, you're right, the analogy is quite leaky if you examine it closely.