Presumably anything that can generate a "high bill" also has some reasonable level of on-call, alerting, SLAs etc ... such that if your Lambda invocations were all taking 30x longer people would notice.
The reason I went for serverless over a server in the first place was so that I wouldn't have to wake up in the middle of the night because of server issues, so alerting is not a satisfying solution. Especially where the problem is more an artifact of how the service is billed than how much it consumes actual server resources.
Good practice for using external APIs is to NOT use any default http client settings and always provide your own timeouts to what you consider reasonable for connections and responses, as well as using a context system with deadlines so you can time out any requests that are taking more than a reasonable time to complete. Making your described surprise long expensive requests into nothing more than short errors (which hopefully you'll pick up on after a while, as long as you've got your alerting system setup right).
I agree, but that doesn’t solve the problem here. The remote API was asynchronous, I was just waiting for it to ack my instruction. Because of issues out of my control (network congestion, maybe?) the time to get a 200 OK from the server shot up.
> always provide your own timeouts to what you consider reasonable for connections and responses
Agreed here as well, and I was providing my own timeout. The problem is that (cost-wise) it’s fine for 1% of requests to hit a 5-second timeout, but gets expensive when 100% of requests do. And lowering the timeout means that during normal times, requests that have latency in the tail of the distribution but ultimately go through would fail, which is undesirable.