You'd be right if the valuations of startups were independent of their eventual market size. In that case, it makes no sense to invest in a smaller company.
But if you're going after a niche market -- a computer vision-based tennis coaching app, for example -- your valuation will necessarily be lower than a company with broad appeal.
So instead of saying, "No, that's a small market" you could invest at a lower valuation and still make the same or higher return as with a unicorn.