There's probably some complications to that, not that I don't think it's a good idea. I say, if so, address them.
For instance heirs. Inventor invents, dies while they would and should still have all rights, say it's the day after being awarded, patent is transferred.
Inventor recieves a patent, but later that asset or all their assets are seized by the government as part of a criminal process, or awarded to someone else as part of a legal action.
There's probably lots of odd situations I can't even imagine where a patent like any other preoperty or asset (or debt or liability or responsibility) is transferred, without it being a sale and without the recipient deserving to be penalized or put under any extra burden.
Even fully business transactions like a company buys an entire other company, rather than specifically a patent. I'm not sure in that case the assets should evaporate or lose their "first sale" extra value. Maybe they should, but also maybe it's just as fair to consider the new owner still the originator, since the originator is a company which still exists. Maybe it should be considered a transfer if the purchasing company closes the original company as a distinct entity and simply ingests the assets.
Maybe this is all pretty easily handled and it's no problem by just having some sort of legal metadata where the patent has a property flag that says if it's considered to be in a state of being before or after first sale, and any legal process that performs a transfer records whether this transfer constitutes a sale or not.
Another gotcha, it probably also means somehow identifying and disqualifying some "gifts" and allowing others.
Anyway I do like the idea.