I feel the same way. I have a 1982 Chevy C10 pickup, and the original owner had put "Flowmaster" pipes on it for a louder, but still natural, sound. With the original engine, it sounded impressive, perhaps more so than the engine actually was.
Not long after I got the truck, the original 305ci engine gave up the ghost (it had over 400,000 miles so it had had a great run) and I had a 350ci engine built for it and installed. None of the pipes were changed, and the exhaust manifold was a factory fit version for that new engine, not custom headers. Still, the sound difference was night and day. Now the truck positively thunders, where it merely growled before. While a lot of its sound can be attributed to the custom pipes and muffler, a lot of it is in the engine too.
To me, it's similar to the difference between a miked acoustic guitar and an acoustic-electric. Either way it's amplification, but the former sounds pure and natural, whereas the latter sounds forced and way too bright, and loses out on the nuances of left-hand technique and right-hand flourishes, as it only picks up the string vibrations themselves instead of the entire ambiance.