To me, the code smell is the syntax from BASIC (and later C) that goes:
<code> x = y </code>
I've always disliked that notation; it looks like the statement "the value of x is the same as the value of y". So "let" turns the code into something that makes sense when read aloud.
I don't think syntactic sugar is a code smell.
let isn't syntactic sugar — it's the only way to declare a reassignable block-scoped variable (as opposed to const, which can't be reassigned, and var, which is function-scoped or globally-scoped).
I'm admittedly not sure what symbol should be used instead. I'd suggest <= except it reads as "less than or equal to".
A lot of JS in the wild favors putting everything in objects (maybe not explicitly, but they end up doing it a lot anyway) so gets away with a lot of "const" use that still lets them mutate the values of those objects. If you're dealing with primitives you're going to need "let" a lot more.