It's a good way to frame the discussion, with the caveat that for some things that subset is 5% of the population, and for other things that subset is 95%.

Is there a threshold? Can we define a principle that covers the entire range?

It seems clear that in the ideal scenario, people's freedoms should not be curtailed merely because there exist other people who would do unproductive things with that freedom. And on the other hand it seems clear that "freedom" to engage or not engage with deliberately targeted highly addictive things is not meaningful, and "individual responsibility" as an organizing principle of society only takes you so far.