You could. But the amount of time it takes is significantly more than yay -S <package> or emerge <package> vs the hell that this poses on Debian (not to mention the dependency hell you can run into)
Gentoo also has the concept of "Slots", so you could have multiple versions of the same libary installed and packages will choose their version to build against accordingly.