To make it big, though, relies mainly on luck and the skill to recognize when luck has provided an opportunity. For example, a lot of these people with passive "affiliate" income are essentially very lucky that they recognized the niche and were in a position to be able to exploit it better than (or at least as good as) anybody else who tried--it's not common, or even 'infrequent'; I'd say it's rare at best.
Or you can do what "everybody" else does: have a job (freelance or otherwise) and, eventually, exploit your connections and network into a product (like PIH's SaaS product) that produces passive income. This, of course, takes years unless as in the above you are very lucky. Even then you have to have some good fortune to maintain steady employment, not be hit with any of a number of life's possible mishaps, etc.
Personally, I think passive income from, e.g. dividend paying equities in the Gambler's Emporium known as the stock market are more likely to provide regular passive income than hoping that working toward some hit affiliate stream or niche software market will payoff without luck.
As soon as one begins to maintain or attempt to expand these things, however, it's no longer really passive. In fact, "building a business" necessarily implies direct action.