Linux doesn't deal well with OOM situations but those don't really apply here. 10 tabs in FF brings me to 3.5 gigs total system use, which is plenty of buffer to avoid swapping and OOM freezes.

I don't doubt you, but maybe your usage was never much memory constrained on either macOS or linux? What sort of software are you running? For what it's worth, I've got an even lower spec machine (4GiB), which works fine for light browsing and similar. But the minimum I need to run smoothly and simultaneously for a work machine is a browser (with plenty of tabs) and some mix of editors and IDEs. Linux craps for me with amounts of RAM that are way beyond what on a mac would still work fine. I don't think there's one single cause for that (e.g. safari is less of a resource hog than what's available on linux, zfs, which I happen to use, is better than apfs but probably also a bit more memory hungry, apple has virtual memory compression by default etc.).

Purely out of curiosity, what do

     ps -A -osize,cmd | awk '/[g]nome/{t+=$1}END{print t/1024^2 "GiB"}'

and awk '$1 == "Pss:" {t+=$2}END{print t*1024^-2 "GiB"}' $(ps -A -opid,cmd | awk '/[g]nome/{print "/proc/" $1 "/smaps"}')

print on your machine?