I think the problem is that we're trying to take paradigms that are designed to be serverful (is that a word?) and shoe-horn them into working in a serverless way.
It's possible and here is the recipe: https://twitter.com/nikitabase/status/1725394843285766463
In the future we are considering separating the buffer pool from compute and turning off compute doesn't drop the buffer cache. It's a research project but PolarDB did that for AliCloud.
Though computers are pretty fast these days (and RDMA exists), so if the pieces are in the same data centre and using good interconnects it's probably not going to make much of a practical difference.