But the patent system applies to a super-set: things that are plain, too.
Therefore, by your own argument, the patent system is operating beyond its justification, and so could indeed be described as 'broken'.
In any case, the patent system is a trade-off -- of increasing production against restricting access (see: the standard economic model of it). If it is badly calibrated, the net effect is negative. And the current state of economic knowledge is that no-one knows whether it is set correctly (see: 'The economic structure of intellectual property law'; Landes, Posner; 2003. Conclusion, p422, s3.). When there is a whole big system, of various obvious costs, and you do not even know the net effect is positive, the sensible thing is to stop doing it (probably by phasing out), until something proves otherwise.