Thanks for the reply! Can you give some examples of these "hero runs"?
No point in staying up waiting for a job, it'd get rescheduled in the early morning at best.
It wasn't the largest cluster around, IIRC 768 quad-core nodes, but I'm sure the meteorological department would find a way to utilize any extra capacity, so still requiring the whole thing all night.
An IBM/360 has laughably less compute than your phone.