Is the the 8GB model? I had the same issue with my workload and found that it was always under high memory pressure, upgrading to a 16GB model fixed it for me.
(If you haven’t looked into SSDGate yet, be sure to check your smartmon to see whether you’re affected.)
https://www.youtube.com/watch?v=FyMCoQmsv-I
The problem is unoptimized apps running on Rosetta2.