Nowhere does anyone claim that "let's go" means "you should let us go". The discussion was whether "let us go" automatically means "you should let us go", which it does not.