> My friends at Replicate told me that a simple rule of thumb for A100 cloud costs is $1/hour.

AWS charges $32/hr for an 8xA100s (p4d.24xlarge) which comes out to $4/hour/gpu. Yes you can get lower pricing with a 3 year reservation but thats not what this question is asking.

You also need 256 nodes to be colocated on the same fabric -- which AWS will do for you but only if you reserve for years.