There are three voiced sibilant affricates in Polish: d͡z, (alveolar) d͡ʐ, (retroflex) and d͡ʑ (palatal.)
The sound of rz and ż is indeed always a voiced (laminar) retroflex sibilant, but and here's the important distinction, following a consonant, ż forms an affricate (the middle one above) but rz does not; instead they remain as 2 separate phonemes (again, dialectal variations may apply.)
The other affricates are dz (alveolar) and dzi and dź respectively (palatal.)
I recommend the Polish phonology Wikipedia page, it seems to me to be relatively well sourced, though maybe a bit difficult to read for beginners: https://en.wikipedia.org/wiki/Polish_phonology#Consonants
EDIT: I think I wasn't clear: there is no difference between ż and rz, except when they follow a consonant. Then Xż is pronounced "in tandem" as an affricate, and Xrz is pronounced as two separate phonemes.
As a Pole too, I'd be against it. There may be no difference in pronunciation, but the spelling is still linked to the grammar and origin of the word.
Eg. while "każe" and "karze" may be pronounced the same, the spelling clearly implies that the former one is a form of "kazać", while the latter is conjugated "karać". It's neither arbitrary nor unnecessary - it's perfectly logical and it conveys actual information.
Eradicating the difference would obfuscate the underlying links and structure completely, and I'm not sure whether it would really make learning easier in the long run. Simplification for the sake of simplification is a lossful process.
*ch and h have historically been two different consonants but have gotten washed down into one.