> yet they allow it because it's their SV buddies.

At one time, Google used to ban sites for "cloaking": offering one version of the page to the crawler, and another to the user.

But over time they got into trouble from these sites, getting sued left and right. Being accused of putting up a wall, and abusing their monopoly. Eventually, these sites won out and Google dropped this requirement.