Let literally means "allow." In many cases where this is said, the person saying it isn't blocking/preventing/gatewaying anyone from going. So the literal meaning of "allow" is not intended.
Let also means "to cause to" as in "let me know", or can be "used in the imperative to introduce a request or proposal", as in "let us pray". (Or "let there be light.")
https://www.merriam-webster.com/dictionary/let
The definition you're referring to matches definition 2a, "to give opportunity to or fail to prevent", or definition 4: "to permit to enter, pass, or leave".
"Let's go" absolutely means "let us go". There's no way around it. It's just not the version of "let" that you may be used to, but that doesn't change anything.
Same as how "let us pray" is frequently used as well.
@ninkendo shared an insightful video below about it. https://en.wikipedia.org/wiki/Clitic
Let us go / Let's go / Let's
If you don't want to use the full form, it shan't stop me.
(Example, "Is this a good idea? Yes, it's!" sounds wrong. But "it's" still means "it is". It would just sound weird to use a contraction in that context.)
Somebody else brought up the example of “let’s go!” versus “let us go” - not the same thing by a long shot.
“Let’s” in English has a distinct meaning from “let us”, and that is to politely and casually (but firmly) suggest a course of action.
I remember touring a Polish salt mine a couple of years ago. The guide was very good, but her English had a few quirks, among them that she seemed to like the phrase “let’s let me to show you …”. It’s wrong, but you can immediately understand that she meant “please let me show you”.