Usually, it's written off as being non-monetizable, too much of a toy to do actual work, "why would anyone do that?", "I don't have any friends that do that", etc.
Search was considered non-monetizable at first. That's why internet giants of the 90's, like Yahoo, were focused on being portals.
PCs were considered too much of a toy to actually do any type of work. Back then, you did serious work on shared on on mainframes.
Radios, when shown to initial investors had one quip, "Why would I want to send messages to nobody in particular?"
When TV was shown to the same innovators in radio, they didn't think it would ever catch on.
Whether large numbers are divisible by primes seems like a useless branch of mathematics called number theory, but we found it useful for encryption. Without it, we wouldn't have online commerce, for math started 2 or 3 centuries ago, when there was no online commerce.
Point is, innovation happens when people try new and different things, even if they're weird. Even if they seem useless. Even if no one else is doing them. Even if people think it's stupid. Even if it doesn't make you money. Because there's something else that's motivating you to explore.
You don't have to be smart to try new weird things, but smart people tend to not worry about what other people think and go ahead and try new things.
Perhaps your quibble is the order of the statement. It's defn true that not all things that smart people do in their free time become something mainstream. But it's defn true that everything that we do nowadays, it was, at one time, something some smart person was already doing years ago.
So if you ever spot someone doing something seemingly weird, strange, useless, non-montetizable, but defn new, then stop yourself from being a cynic because it might be something you end up doing yourself in 10 years, as improbable as that seems now.