Conversely, open source doesn't necessarily mean "free to use without encumbrance." There are many open-source licenses that forbid certain uses (e.g. Business Source License). On the other hand, a work in the public domain is free to be used by all without restriction.
A better analysis of open source vs. public domain would be in the form of a square, where one dimension would be the right to use the work, and the other dimension would be the ability to obtain and contribute source code.
Also, many licenses, such as the GPL (one of the very first "open source" licenses), have certain encumbrances; you cannot redistribute GPL-licensed software without either including its source code or making it readily available.
In other words you don't use PD software "like Free Software". You can use it in many places where Free Software would not be permissible.
In terms of -developer- freedom, public domain is top of the pile, the Open Source, then Free Software.
In terms of -user- freedoms Free Software is top of the pile, OSS in the middle, public domain is similar to commercial software.
GPL is copyleft. MIT, BSD etc. are permissive. But all of those are both free software and open source, which are essentially synonyms.
The reason so many people get confused by this is that some of the people who prefer copyleft licenses (notably the FSF) also tend to prefer the term “free software”, for philosophical reasons.
In your analogy we're not the ones calling elephants bananas, you are. We want to keep calling one bananas and the other elephants. You are suggesting that since elephants are similar to bananas you can simply use either word.
Legally, Open Source and Public Domain are -very- different animals. Open Source comes eith a copyright, and a license (which has requirements), public domain does not.
Of course public domain and open source are both "shipped as source code". Then again so is a fair bit of proprietary software. That doesn't make it open source either.
Similarly the common use of "open source" is different to the OSI's preferred definition. Note that the OSI's preferred definition is not a legal definition. It's just what they prefer.
Please note that public domain laws vary depending on the country. What you call a banana might mean something different elsewhere.