I don't have any solution for this problem, but one thing to note is that if you set the appropriate value of work_mem in the session that runs the parallel query, the same will be used by parallel workers.
The problem I see with this approach is that you don’t know when the planner is going to decide to use parallel scans meaning that you might get worse performance for non parallel queries.
That's right, but you can manage if you know which query is going to take parallel plan (using explain). You can set it before that query execution and reset once query is executed. I understand that this can't be a generic solution, it can help only in a limited set of cases.