You can either spend 4x the amount of time to make them compatible (and still follow TDB) OR spend x time and just deliver to each customer the respective feature only (and no longer follow TBD)
Also this process can continue. In the next iteration you have will 3 incompatible features with the same or different customers.
If you use different branches you can always to go to branch X and know exactly what is deployed on customer X.
In your case the only way to recreate what customer X has is to find the proper combination of libraries that you have shipped to them so far.
Imagine I am a developer and want to do a hotfix for a specific customer. I can just do git checkout branch-of-customer and start working. Isn't the process more difficult with your approach?