> This sounds like a failure of business model rather than an intractable problem.
Not really, the business model is highly successful, even in the face of rampant piracy. The free software movement might consider it a failing of the business model, but unwinding that would mean applying the free ideology to all aspects of copyright and trademark, and not just the free software movement.
Moving past that, your hypothetical has some deeply flawed considerations.
First, you cite an open source game, not a free game. It is easy to sell open source software that is not free, and this is a well established fact (look at the id Software comments below). Also consider Oracle, which sells a large number of open source products, as does Google, and to a lesser extent Microsoft. Their open source offerings are offered under free licenses, but are backed and protected by extensive IP, including trademarks and patents, which essentially makes the products non-free.
Second, setting up a build environment and building the software is more technically challenging than pirating a copy, however, much like pirating a copy, that technical cost only needs to be paid by a single user who is motivated to share the software. I submit the entire history of the software protection mechanism / cracking scene arms race as evidence that such a cost will readily be paid.
Third, shifting the discussion back to free software, the sale of open source software is usually predicated on a license that prevents other sellers of the same software from competing on the consumption of that software - whether it is a trademark, or proprietary bits that are compiled, or hardware that requires that packages be digitally signed, or any other number of constraints. This even applies to GPL software - the original owner of the GPL content can sell a copy of the code under a different license, however anyone who forks or extends the GPL licensed code is obligated to propagate that license forward, which severely hinders a competitors ability to sell the software.
For software to meet the constraints of being free according to the FSF/GPL, these restrictions cannot substantially exist as the licenses provided allow a user to remove the technical constraints, or replace the trademarked content. It is certainly possible to release the software for free and retain control of the trademarked/copyrighted elements (for example, Quake 3, Doom 3, etc).
Do note that the companies that have released their game engines as free software have typically done so as a marketing move to allow developers and engineers to learn from their code / build on their code, and still offer commercial licenses - again, ancillary to the core release of the game. These releases rarely coincide with the release of the game since, again, the primary gain is capturing revenue from the initial consumption of novel content. I am not arguing that Free Software is bad, I am simply saying that the expectation that all software must be free is an untenable position given the cost (and associated risk) of developing some classes of software.