Former green card holder here. A green card (or a visa) gives you a privilege to entry, not a right.