Right, perfectly reasonable. I'm still sort of on the fence.
I guess where I would respond though is: Most of what you're talking about are singular rare-ish exceptions. We're talking about a very common staple here (which means, yes, there will be confusion of some sort, perhaps in baking?)
I think my reasoning here is, we're not calling margarine "plant butter," we're not talking about "tapioca/potato eggs" (had to look that one up.)
Overall, even, I feel like things would go smoother if we used some other word, e.g. I know "Silk" is trademarked, but something along those lines.