1) Cost. They entered the market at an extremely high price point. Actual interchange rates on swiped cards are 0.05% (for most debit/check cards used as credit, since 2010) to 2%. Square charges 2.75%. Any business that currently takes Square can add 1-2% to its margins by talking to the nearest bank about a merchant account, which they've likely offered since the 1970s.
Now, merchant accounts typically come with some other costs ($20/mo or so in fixed fees, different rates per type of card accepted, more expensive terminals/hardware, etc). But if you're not a food truck or flea market vendor, your processing volume will cover those cost in no time, which is why you don't see Square outside boutique stores.
2) Ease of use. Square's dongles and apps are much more difficult to use than a dedicated card terminal. The customer doesn't get to swipe their own card, the store doesn't get to accept debit at all (which would be MUCH cheaper for them), and they're really finicky to actually get a good swipe on. Their POS software is a toy version of real POS software; it's got maybe 10% of what you need to run a retail outlet, which is why its use is limited to vendors with a catalog that fits on one or two screens and who move little enough product to do their inventory and accounting with an Excel spreadsheet.
3) Ease of signing up. This is the only place Square actually competes (and wins with a certain audience). You pick up a free dongle, install a free app, and you're in business. If you'd like to add 1-2% to your store's margins, you take 3-5 business days to fill out an application and credit check for a merchant account, wait for approval from an underwriting department, then buy POS software and a card terminal. It costs a little more up front, it takes longer, and it's intimidating. That's where Square shines.