Suppose X is sandwiches. If I'm allowed to charge anyone who makes sandwiches, yes, my patent is worth lots of money. It's not "worthless" in that sense. It's worthless in the societal sense: allowing my monopoly provides no value to society.
Nobody thinks you can't make money from software patents. Some people think software patents do not provide value to society by encouraging innovation, which is the stated legal intention of intellectual property law.
A (neo-)liberalist would not approve the difference, as long as money is made, but the patent system is government regulation of trade, so he would not approve of the patent system itself if he took his conviction to the logical end.
When people call patents valueless, they're probably talking with respect to society as a whole.