Depending on the image context it can also backfire, though not so much for photos, but rather screenshots. E.g. sometimes a 2 color dithered image results in a much bigger PNG than, say 16 colors without dithering, because bigger areas of uniform color compress much more efficiently. So I usually just try "everything", to find some sweet spot between file size and quality.
Is there a tool that allows you to select which area of the image to use what dithering parameters and color count for, with a live preview? I know I'll vibecode one myself just to see what it looks like, but maybe this already exists in a "real" application? Then you could e.g. use dithering on the faces, but for the background just a few colors without dithering.