> but then you're not directly selling your software either
You can't usually sell FOSS because any of your downstream customers can undercut you. Since they may be doing less development than you (to the point of doing nothing), any business model around FOSS that relies on directly selling your software is risky. I suppose it can work, but it requires all of your downstream customers to be working in concert with you, which is very unlikely long term (and especially unlikely if you start making a lot of money).
You can try to offer SaaS services on top of your FOSS software, but that will also be undercut if you are reasonably successful.
This gives you a couple of options. First, you can try to eke a living out of selling software/SaaS but making little enough money that you don't attract attention from competition. This is not necessarily unreasonable if you are trying to make a living wage. People are unlikely to compete with you if the pool is only $100K a year.
But in reality, the only actual scarcity that you have to offer is your time. Very rarely do you see someone fork a project, do a huge amount of development work and offer up the result. People/organisations trying to cash in on your work will be trying to avoid doing development. Otherwise they will almost certainly prefer to look at your effort and rewrite it from scratch.
This means you have to charge for development work. There are only really 2 ways of doing this: find contract work for your project (up to and including sponsoring development entirely), or asking for willing support of development (aka "donations").
I think it's incorrect that donations aren't enough. It's fair enough to say that they generally haven't been enough, but I think part of the problem is the idea that a "donation" is optional. Especially for business software/services, one thing I've heard over and over again is: "We would pay for this, but there is no way to get an invoice". IMHO, this is actually the problem. It's not that there is a shortage of people willing to pay, it's more that there is a difficulty making the payment happen. Even non-corporate end users are willing to pay for software and support development (what percentage of people would hold up their hand and say they wouldn't pay for the tools they use every day). It's more that they don't pay (strangely, quite a few free software game developers have discovered that putting their game on Steam results in money appearing, simply because it is a convenient method for people to pay).
I think part of the problem with current FOSS business models is that they are predicated on the idea that people won't pay unless forced to rather than the idea that people will pay, but don't because there is too much friction in the payment area (psychological as well as actual).
Fix the payment issue and I think you will find that you have a viable business model which is robust because anyone wanting to challenge you has to produce better than you, rather than just ride your coat tails.
One small thing I would also advise: don't overlook trademarks! There is a very good reason why the GPL does not include a trademark license. Do not hand out your trademark to your downstream customers and allow them to piggyback on your success. Force them to build their own brand if they want to market software/services using your code.