pg has a fairly clear set of factors that he thinks indicate success -- being physically in a startup hub, for instance; having multiple founders; the sort of thing he recommends in his essays. Rejection from YC can mean that they don't think the project can succeed; it can also mean that they don't think the project, even if it succeeds, will give them the kind of return on investment they want.
So (to make the math easy) suppose they have a company that they own 10% of, that they've invested $15K in. This means that for them to break even, the company has to eventually have a value of $150,000. A rule of thumb for business valuation is that the sale price of the business is 3 to 5 times its annual earnings -- which means that the business has to earn $30,000 a year to $50,000 a year in profit for that to be worthwhile.
Now, a micro-ISV, for instance, that treats its founders' salaries (appropriately) as expenses may be a terrifically successful business from the point of view of the founders, especially if they pay themselves acceptably well and like the working conditions. But unless it makes $30,000 to $50,000 in profits a year, it's a losing proposition for Y Combinator.
There are other ways it can be a winning proposition without those profits -- if it's just before the tipping point of a big trend, or if the founders are especially skillful. A company that isn't making any profits at the moment but has the potential to make really big profits, or the potential to be sold for a lot of money, might be a worthwhile investment.