Would this theoretically be possible with femtocells and phone signals?
You can do this with just receive though. A $10 USB TV Tuner and a Raspberry Pi will listen in to most cellular bands, and pull enough out of the over-the-air machine to machine chatter to do a similarly accurate job of counting cellphones...
The cellular transmitters (can)_ run with a lot more power than Wi-Fi, and are on lower frequencies, so the range at which you'll detect them is significantly longer, which might make localised device counting less useable.
But then you are not in the price range anymore.
Retailers rollbacked to IR sensors and now are slowly adopting counters on cameras with deep learning for disambiguation.