Although there are exceptions to it, eg. "budżet" (or budget) is typically pronounced as if it were spelled "budrzet", apparently owing to the French origin of the word.
As a Pole too, I'd be against it. There may be no difference in pronunciation, but the spelling is still linked to the grammar and origin of the word.
Eg. while "każe" and "karze" may be pronounced the same, the spelling clearly implies that the former one is a form of "kazać", while the latter is conjugated "karać". It's neither arbitrary nor unnecessary - it's perfectly logical and it conveys actual information.
Eradicating the difference would obfuscate the underlying links and structure completely, and I'm not sure whether it would really make learning easier in the long run. Simplification for the sake of simplification is a lossful process.