I have an M1 pro (16gb) and don't have this issues. Might be worth checking your system/taking it to Apple for a check.

Only time I got any similar issues were when I was hitting the ram pretty hard, but MUCH better than my i9 32 Gb with the same problem on memory (FUCK YOU DOCKER!)