Sounds like you’re looking for T-digests[1] - most production systems I’ve worked on that do this are using them under the hood.

1: https://github.com/tdunning/t-digest