This is the lump of labour fallacy. It's largely regarded as fallacious now; there's a decent Wikipedia article on the topic if you're interested.