I can see your concern with GPL-family being like a virus. Indeed, it is very restrictive and contributors may not want to get involved due to the logistical overhead of trying to add features to/improve it.
Addressing your concern about not being paid for your work, this is an age-old problem with open source. It's hard to make money off a product that is freely available to download. A good rule of thumb is to make users feel like they _want_ to give you money, instead of making them feel like they _have_ to give you money.
Some options you have if you want to keep it open source but still make money:
1. Do the dual licensing approach, and use one open source license of your choice, and one commercial license to allow businesses to use your engine in their proprietary software.
2. Donations link in the open source software
3. Paid technical support/Freemium model
4. Paid training programs to train users to become an expert with your engine
5. Early access subscription model: create a system where users can pay a monthly subscription fee in order to get new features earlier than official release.