You can beat AMP's performance with hand-tuned optimizations. Its goal is to uplift performance across a significant percentage of web content (instead of just that developed by the few experts).
Is Google going to penalize my sites if we don't use it? No, AMP is not a ranking factor.
We don't include advertising in our pages, so should we care about AMP? Advertising is to AMP like Advertising is to the Web. E-commerce is a big use case where advertising is uncommon.