The most common, plain meanings of the phrase “rightfully so” do imply a basis in morality.

It’s fair to say that it can mean that the “rightness” is derived from other sources, but pretending that the primary meaning of the phrase doesn’t imply morality is just being intentionally obtuse.