Ask HN: How to deal with GDPR / cookie notices in the context of a crawler?

(For background, I was responsible for figuring out GDPR compliance in a past job, so I've picked through the literal text and a lot of interpretations of it.)

I think there's a spectrum, but I think the vast majority of the cookie notices I've encountered have been implemented in a sneaky way that runs counter to the spirit of the law.

The spirit of the law is that sites should explain how the data is used in a way that the layperson can understand, and it should be clear to the layperson that (in most cases) the site is legally obligated to give you a way to say "no."

As it stands, most GDPR notices give you a choice between "OK" and "more info" (where the "no" option is hidden) or between "Yes" and a subtle X in the upper left (because upper right would be too obvious). And they don't tell you that by clicking "Yes", you are consenting to having your information brokered and sold to innumerable advertisers.

I think that's a dirty UX trick, and that for the purposes of getting consent the "no" button actually should be an obvious "no" button.

The reason the badge isn't possible is that GDPR did the right thing to enforce privacy by default, and all of the sites that want to monetize your data for advertising have to get your explicit consent. So you get all of these notices because they have to and want to ask you. Were it not for advertising, you GDPR would have been a pretty peaceful transition with a few exceptions like "oh yeah we keep crash reports, you're okay with that right?"

Given how hostile some content providers are towards the GDPR I always assumed pressing X means they will violate the law and store my data anyway.

I mean, most of them already violate that clause where consent for advertising or tracking should be opt-in, not opt-out. X implying consent is just an extension of that.

Actually it's really hard to argue informed consent with this. The regulators could easily slap them for that, if they'd want to.

Add to it that most sites make it impossible for you to use the site without enabling cookies or if you dont consent to them storing cookies.

GDPR actually draws a reasonable line for this. Using cookies to remember your login or browser sessionis fine, and you can probably not even ask to do this. Analytics is a bit trickier, but as long as the data is aggregated and no "personal data" is collected, you're on the safe side of the gray area.

The problem is that using cookies for advertising does require you to ask, so how do you conflate the two?

I've seen some sites phrase it like "we use cookies to personalize your experience". You can interpret that to mean session cookies, but you can also interpret that to mean marketing.

I hope EU regulators end up actually going after people for doing sneaky crap like this.

Indirectly the solution/workaround the sites have come up with is saying, "if you want to use our site, you will have to agree to our cookie settings. Else don't use the site."

I have cookies disabled by default. Whenever a site does not work without cookies (news sites, travel sites, and blog sites especially), I open them in a guest mode. Still feel, its a bit tedious thing to do, but works for me.

The irony of course is that GDPR actually forbids this sort of "all or nothing" deal under specific circumstances. But it's a gray area, so sites just say "accept how we use cookies."

Oh well, it's still better than before.

I find them to be pretty much as useful as the ubiquitous Proposition 65 warnings (ubiquitous to California, that is) placed in almost every business. Similar to the prop 65 signs that tell me living in the modern world exposes me to certain chemicals that carry a non-zero risk of harm, these GDPR/cookie notices essentially say using the web will expose me to cookies and such... in both cases, I already knew it and the downsides seem to be worth all the upsides.

Having said that, Prop 65 warnings usually sit quietly on some wall or paper somewhere, annoying only the business owners that have to print/place them, whereas the GDPR/cookie warnings thrust themselves in my face every damn time I visit a complying website in a new session. Very annoying in a way Prop 65 warnings are not.

The GDPR notices are supposed to make it as easy to opt out as to opt in, so they shouldn't really be warnings.

The problem is websites have apparently decided it's safe not to comply with that portion, presumably knowing everyone would opt out.

You can always "opt out" by not using the web site, unfortunately.

So I don't know that they would lose any challenge on those grounds.

That doesn't count. You can't refuse service (or provide worse service) because someone doesn't consent to unrelated data collecting / processing.

> Overall I feel like the intent of these is correct, but the execution is terrible.

The inverse is true. Instead of a simple "opt in" / "opt out" dialogue, most sites are deliberately employing various UX dark patterns to coerce users into agreeing to tracking against their will. These range from the annoying: settings -> untick several checkboxes, to the plainly non-compliant: accept all tracking, or leave the site.

I hope that enforcement gets into gear soon, and takes a very dim view of these practises.

I hope that the website publishers will see that nobody wants to be tracked and that the advertisers will find out that tracking is pretty much useless anyways. A website that only uses cookies for logging in or storing settings does not need to have a cookie notice! In a similar vein, you do not need a GDPR notice if you do not track your users and only store anonymized, non personal data.

To me the publishers are currently grasping at straws. Also, much like it is a complete failure of design to ask permissions for all of your stuff in a mobile app at the beginning, it is so with websites. A GDPR consent could as well be integrated into the signup process.

I doubt that EU will prosecute companies for this, but to me they are clearly acting in bad faith towards one piece of legislation that actually tries to somehow protect peoples privacy.

> advertisers will find out that tracking is pretty much useless anyways

It is only useless for those with at least half a brain. For the rest, it does kinda work. Sure, the gains are tiny, but when you're desperate (which advertisers are, because otherwise they would've taken a more respectable career path building products people actually love instead of spamming the world), every little bit helps.

The real solution would be better user education to make them distrust advertising (because if a product is so great then why would someone pay money to put it in front of you?) and then this cancerous industry will die off by itself. There will be no more incentive for tracking because even the best tracking in the world won't help when your target market immediately sees you as a scam.

Personally I am not that against advertising. At some point people have to learn about your product by some means. The offer is just huge nowadays.

Personally I have build a very good product (got great reviews on a few respectable sites and above average conversion ration) but in the end I am quite desperate because it is actually very hard to get people to know about it without an advertising budget.

I love them. I use ad blocker to remove the notices. This way I have not agreed with anything they want me to.

If they will track me, they’ll be breaking the law, and we both know it. And I can access the sites just fine.

Then visit fortune.com, it’s one of those sites.

Opens OK. Adblock status counts 1-2 adblocked things every second I stay there.

If it’s different on your PC could be ‘coz I’m in Montenegro, it’s Europe but we are not yet in the EU.

I wish I could just toggle an option in Safari to automatically reject tracking and begone with it. (Then again, my understanding is that some websites refuse to honor "Do not track" so what hope is there for this to get implemented let alone honored?)

> I'd much rather have say a badge in the address bar of the browser (similar to the https badge) saying a site was gdpr compliant and used cookies

This would also be much easier to implement. Just add a GDPR-Compliant header (or similar), and the browser already knows if there are cookies.

I think the actual issue is that browsers are open source/non-commercial things that are usually owned by companies in the US (a place where EU laws do not directly apply). They would have to ask nicely instead of making it a law.

I look forward to a future where the web has separate <ad> <tracker>, and <drm> tags. Would standardize all this garbo and make blocking components much easier.

Can anyone just write an RFC?

I love this. And Europe can make these tags mandatory for all ads and trackers.

I wouldn't like to see that. What's an ad, what's a tracking script? Regulation like this usually has unintended consequences.

I'd expect the push to come from browser vendors. Either by blocking ads that aren't in those elements by default, or feeding them garbage data.

There could be a technical definition of an ad:

"user-visible text or image(s) meant to advertise a product or service, separate from the primary purpose or offer of the page in which it is embedded"

... or something. IANAL.

For an easy solution, you could leave it up to browser developers. More complicated: anything you were paid to put on your website. Exceptions for integrated advertisements/ads embedded in videos.

This won't really hit people doing self-serve or custom ads much, but it'll cover the vast majority of the web.

The issue is the law requires no tracking by default, and that it be as easy to opt out as to opt in.

Unfortunately sites have decided not to comply with this portion.

About the same as I felt about popup ads when it was a thing. Eventually these will be reliably filtered by ad blockers I guess.

In more philosophical terms, how am I supposed to feel about profit-optimizing websites that I don't want to pay for choosing to be shitty to me? Meh.

Is there a project like that or should we start a gdpr blocker? Let's say it will have a database of websites with a way to block the popup on each of them.

I'd welcome an extension that would mark links to websites that block EU web users. It's quite annoying to click links only to be told "fuck off".

Anyway it's not so simple, as links are not what they used to be anymore either on many websites. All those shorteners and opaque tracking redirects... bleh.

There's https://easylist.to – might want to start a new list in this format or look for existing ones on github.

I share the same sentiment (and yes, it seems that it's "broken for good" - i.e. it will stay indefinitely).

There is https://www.i-dont-care-about-cookies.eu/ project. Still, it's the most annoying on phone, where I see now solutions.

If the website is forcing me to agree to continue then I close the tab, I hope many people does the same and the publishers notice that we care.

I think they are useless when there is no "don't drop cookies on me" option.

more like "Hey we have to tell you we're doing this evil thing, ok?"

Obviously the 'negative' response is to hit the back button on your browser.