I sometimes think tomatoes are all born with a fixed amount of flavour in them, and the larger they grow the more dilute that flavour becomes. Because I agree - the little ones are fine.
Similarly, canned tomatoes taste great[1] because they are picked ripe and packaged.
[1] Canned tomatoes have a slight cooked taste so they're never going to taste fresh, but are great in anything you're going to cook.