The problem I see is that they both, especially when used 'improperly', prevent people from innovate.
To me it looks like software improves also by recycling and improving existing concepts so, when it comes to software, patent owners eventually manage to raise some money, but that's all.
They cause much greater damage to the rest of us by stopping other people from reusing the concept.
The thing is that from my perspective, the same happens with most of the grants given to copyrights holders.
Again, when speaking about software, to give a copyright holder the right to be credited for the work is perfectly fine, but what about the right to determine who may adapt the work to other forms, who may perform the work, who may financially benefit from it?
How different is that from a patent?