One way I deal with this problem in an alternative workflow manager (Nextflow), is by calculating the memory requirement for the ~95th percentile of a job, and submitting with a rule "If this crashes from going OOM, re-submit with memory*N" (up to some max number of retries/RAM). This lets most jobs sail through with a relatively low amount of RAM, and the bigger jobs end up taking a bit more time and resources.

The better your estimator function, of course, the tighter constraints you can use.