Audiophiles usually recommend using some better algorithms such as SSRC resampler (foobar2000 have it as plugin handily) than whatever FFMPEG provides.
I can't say I personally can tell the difference, but based on my (limited) knowledge of DSP, artifact-free resampling definitely isn't a straightforward thing.