For me it's not about trying to nail down sentience but rather a basic heuristic about the creature's ability to feel pain. If I cut it or take its babies and it produces a response that indicates it doesn't like that, then I don't eat it. Fruits easily pass this test because the plant encourages us to take them (although I recognize that we've gamed that system and it's not like we're hunter gatherers anymore accidentally spreading seeds for the plant's benefit). Vegetables less so, but the response is nowhere near as visible as mammals and birds and fish and even bees. I recognize that pain is also a squishy, debatable concept but personally this is a situation where I don't think we'll have a rigorous intellectual framework in our lifetime so we just have to trust our intuitions; which is why there's so much disagreement (your intuition might be the opposite of mine).

I'll also mention however that I'm not solely vegan for ethical purposes so my conviction in veganism doesn't hinge on being perfect here. When I realized the health, environmental, and spiritual benefits of veganism, in addition to the ethical angle, it was a pretty easy decision to make.

But to your point, yes I do think we (everyone, not just vegans) are biased towards bigger creatures that we can see with the naked eye.