Abbreviation does change it; it narrows the meaning. "Let's go" never means "you should let us go" and "let's pray" never means "you should let us pray".

@ninkendo shared an insightful video below about it. https://en.wikipedia.org/wiki/Clitic