In general, the market tends to solve the "you've done one small bit of coding" problem; if a proprietary product is only epsilon better than the open source version then people will only be willing to pay epsilon for it. Conversely, if people are willing to pay real money for that small bit of coding, then by definition it must be a valuable bit. (It's even possible to make money by selling unmodified open source with some slick marketing, which really tends to annoy hackers because it implies that the marketing is worth more than the code...)
Also, "you must provide your source code modifications back to x264 LLC" http://x264licensing.com/faq