There are three voiced sibilant affricates in Polish: d͡z, (alveolar) d͡ʐ, (retroflex) and d͡ʑ (palatal.)
The sound of rz and ż is indeed always a voiced (laminar) retroflex sibilant, but and here's the important distinction, following a consonant, ż forms an affricate (the middle one above) but rz does not; instead they remain as 2 separate phonemes (again, dialectal variations may apply.)
The other affricates are dz (alveolar) and dzi and dź respectively (palatal.)
I recommend the Polish phonology Wikipedia page, it seems to me to be relatively well sourced, though maybe a bit difficult to read for beginners: https://en.wikipedia.org/wiki/Polish_phonology#Consonants
EDIT: I think I wasn't clear: there is no difference between ż and rz, except when they follow a consonant. Then Xż is pronounced "in tandem" as an affricate, and Xrz is pronounced as two separate phonemes.