16×16 and 32×32 are far more important as they’re used far more—most notably, tab bars normally show icons at 16×16, which is commonly 32×32 on high-DPI displays (assuming a scaling factor of 2×). And those small sizes are also where you want to be the most careful about the rendering.
I like to craft precise 16×16 and 32×32 icons, and then generally jump straight up to 256×256 or 512×512 and let anything that wants larger than 32×32 downscale from that. (I used to go for /favicon.ico containing 16, 32 and 256, even though that bloated the file size by a handful of kilobytes, but now I’m more likely to go for /favicon.ico containing 16 and 32, and a PNG containing 256 or 512 or perhaps just shifting to SVG at this point.)
Fun fact: just as the ICO format lets you have completely distinct images for distinct sizes, an SVG file can contain multiple images for different sizes by the use of media queries; for example, you could show a tweaked icon at 16×16 and another at 32×32, and something else at all other sizes, while omitting some tiny detail below 64×64, and replacing the icon with icon-and-brand-name above 128×128. Just be aware that the media query will be matching device pixels rather than CSS pixels, which may or may not be what you’d like.
Caution should also be exercised with the prefers-color-scheme media query on SVG icons. It does not reflect whether the icon is being displayed on a light or dark background. (prefers-color-scheme: dark) hints that your icon is probably being displayed against a dark background, but it could easily still be being displayed against a light background; and with (prefers-color-scheme: light) it will be quite common for the icon to be being displayed against a dark background. You need to carefully design your icon so it works well against almost any background colour.