Thanks for sharing that experience. What did you end up doing to lower your costs?
- We used gRPC with no compression before, and we tune an acceptable compression level to achieve a balance of performance and data volume.
- Improve our SQL optimizer to consider the network cost more accurately and data placement rule to balance data more logically to reduce data transfer crossing AZ.
- As a cloud service, we will also expose the cost transparently to our customers like MongoDB Atlas does. :-)
Suffer and try to vote with your wallet (example oracle cloud has lower bandwidth pricing).
One of the real problems is that cloud providers can just waive cross-AZ traffic for their own managed services but charge a lot for cloud customers.
That is why no matter how smart your software is they'll always have the upper hand and build things better themselves (why everybody using the BSL license).
It’s like owning the entire supply chain. I remember this bit from someone who used to work at Heroku. It’s impossible for us to switch we spent so much time making the system work with AWS, switching or adding another provider would be a kin to starting over.