Went down this road recently and I ended up deciding to work with holoviz instead: https://holoviz.org/

Having some level of abstraction on top of matplotlib is necessary if you don’t want to spend time tweaking every little thing.