This is doubly true if you're buying button-down shirts, because if you're not one of a handful of exact anatomical sizes you've got to have things altered after purchase if you want them to fit well.
Men's button down shirts are generally sized in 3 dimensions - collar, sleeve and fit (slim / normal)... which is more axes than most clothing items. If you're only seeing a handful of sizes, find a different brand.
Unless you're trying to buy a flannel. Then it's small medium large.
You can get custom sized flannels too, I'm wearing one right now. Just have to look around or pay more but it's worth it if you're a fringe size. (I'm tall.) I get mine from Trumaker.