Yeah, it's not clear if a self-landing retrobooster would have been possible given the components available at the time of the space shuttle's design, not to mention the vast improvements in computer modeling and simulation that we're available to SpaceX.

But, of course, that's only part of the story. The space shuttle had additional political constraints, i.e. spreading the wealth to many congressional districts, that greatly increased the cost and complexity.