Google vs. the Open Web
interpeer.io
interpeer.io
Is this serious?
Speaking as someone who works at Google, but not on anything at all related to ads, browsers, or ad spam detection, I only wish that the attackers who (try to) make a living out of scamming Google and its advertisers out of money were as incompetent as the author of this article appears to be.
There are two kinds of bots.
There's legits ones that the site owners will generally find to provide a positive tradeoff. These bots identify themselves by the user-agent, the requests come from a predictable set of IPs, and the they obey the robots.txt. Think most crawlers for search engines (though not Brave's), bots that handle link previews for apps like WhatsApp, even RSS readers!
Then there's the abusive ones. These are usually hitting resources that are expensive and contain valuable information. The will not obey robots.txt. They'll run residential IP botnets to avoid IP blocks. They'll make their bot as similar to legit traffic as possible, the user-agent is literally the first thing they'd look at changing. They'll hire mechanical turks to create fake accounts to look like signed in users.
Now, it's pretty obvious why the author's methodology for supporting the statement is so silly. First, it was circular. They identified bots by user-agent, and then declared that since there were bots that had a distinguishing user-agent, the other traffic can't have been bots. The other is that they looked at the logs of a server that doesn't contain any data that somebody would be scraping maliciously. Ocean's 11 will do a heist of a casino, not a corner store. Likewise the professional bot operations are scraping valuable information from people trying to actively defend against it, not your blog.
i also found this odd. the target matters.
when i did more sophisticated dual tracking (js interaction and http logs), the js interaction detected 2x-3x the bots vs pure logs from just UA strings and ip ranges.
Instead of simply flailing our collective arms around complaining about an evil corporation, has anyone written to the respective competition authorities (such as the FTC in the US or CCI in India) about the potential anticompetitive effects of this proposal?
Their leadership isn't "awful", their leaders are finally doing their job, for the first time in decades.
What they lost in the case they filed in June was an enjoinment to prevent the merger and acquisition of Activision/Blizzard until their own FTC judge (read: an administrative law judge that exists outside of Article III and is within the chain of command of the Executive branch) could hear the case on August 2nd. The merger had a termination date of July 18th, so they needed that to continue their administrative review. Discovery was finished, it was just the trial, but without being able to enjoin the trial because in the opinion of Judge Jacqueline Scott Corley they were unlikely to prove the merits of their assertions, the trial before the FTC judge would have been moot by the time it occurred. It’s been formally cancelled by the FTC by the way.
I’m not disagreeing with you by the way, I just wanted a place to park this information in the discussion. They started this action in December and failed to win even an enjoinment against Microsoft and Activision temporarily stopping the merger until their own guy could hear the case.
But that doesn't mean that what they do, big picture, is pointless.
I've also created a parliament petition, which has gotten the 5 min supporters it needs before they review and publish it. I will share it on HN once its published.
Edit: removed the link to the petition for now (it'll come back after its published)
- ban Google all together in your personal life. No chrome and no excuses. Stop the bullshit or leave this profession. Use startpage, duckduck or whatever for searching.
- develop with and for firefox and friends only, introduce usability problems for chrome
- employ the same tactics as google.
-> Bundle firefox with the software you are distributing.
-> Like google did, remove the competition altogether from the users device.
-> make your npm-module or your website slower in chrome
-> let your customers know that your service for non-chrome users is cheaper. Money motivates.
-> show a popup urging users to download firefox, provide a link to download or page with more explanation.
Tell that you detected that their current chrome has security and privacy risks and that you recommend to take action immediately. Average user is easily scared into action.
-> use as many tricks as you can think of to spoil the well for google.
Destroy search results, fill their storage with /dev/random, whatever your imagination leads you too. You keep telling us you are so smart. Show it.
If I recall correctly, this was Google's approach with Chrome.
-> let your customers know that your service for non-chrome users is cheaper. Money motivates.
This is incredibly hypocritical. I would never want to work with or for, or employ, or be friends with or associate myself with someone who blatantly displayed this level of hypocrisy and lack of integrity.
This comment is also clearly violating the HN guidelines - it's not intellectually interesting, it's naked political activism.
On hypocrisy: Both cancer and cancer surgery are aggressive, doesn't make them the same. Google violated competition laws, we as individuals don't.
Has anyone already made a template for this that you can easily include in your website?
Contact info for antitrust authorities:
US:
- https://www.ftc.gov/enforcement/report-antitrust-violation
- antitrust@ftc.gov
EU:
- https://competition-policy.ec.europa.eu/antitrust/contact_en
- comp-greffe-antitrust@ec.europa.eu
UK:
- https://www.gov.uk/guidance/tell-the-cma-about-a-competition...
- general.enquiries@cma.gov.uk
India:
- https://www.cci.gov.in/antitrust/
I could not find an easy contact method for filing a complaint for the CCI, but it looks like this is the process?
- https://www.cci.gov.in/filing/atd
Canada:
- https://www.competitionbureau.gc.ca/eic/site/cb-bc.nsf/frm-e...
I'm happy to share what I've sent to the FTC if others want to use it as a template.
I would like to bring your attention to Google’s recent proposal to add a feature to its Chrome (Chromium family) of browsers called Web Environment Integrity. This provides a mechanism to reinforce Google’s already dominant browser market position by creating a technological control that can be used to nullify a user’s choice of browser, device and operating system. This technology also has the potential for abuse by preventing users from using browser extensions that can enhance security by blocking unwanted and potentially malicious content, as well as browser extensions that help vulnerable users with enhanced accessibility needs, such as color blindness and visual impairment.
Google’s dominant, near-monopoly position in the browser market already harms me as a consumer by reducing browser choices and preventing a competitive market for developing new browsers. Allowing Google to include this feature will reduce my browser choices and consolidate the browser market even further, and it is incumbent on [INSERT AUTHORITY HERE] to take action against this abusive behavior.There were some reasonable criticism that it was too long/wordy hopefully you can adapt and reformat to your liking.
Dear [Recipient],
I'm writing to highlight an important matter regarding the digital freedom and competitiveness on the internet. Google is rapidly advancing a policy named "Web Environment Integrity" (WEI) in their Chromium browser.
WEI allows developers to regulate browser configurations, which could lead to limiting the usage of free browsers or operating systems. This creates a potential for a web environment that discriminates based on browser usage. Further, this scenario could pave the way for governments and corporations to enforce specific browser usage and could also allow Google to restrict access to their services based on browser compliance.
This practice contradicts the fundamental principles of an open and competitive digital marketplace. I strongly encourage your agency to investigate the potential impacts of Google's WEI and consider taking necessary actions.
Your proactive engagement is vital in preserving the principles that ensure a free and open web.
Best regards,
[Your Name]
Yes, I have. A couple times now.
Google has been strongly signaling this since last year. No one wanted to believe it last year though, before the tech bubble burst. Now that people see Google isn't so awesome right now, perhaps more people will write and contact their representatives.
Just a reminder that several states have already filed an antitrust suit (in part) over a previous Google plan to turn the web into their own walled garden.
> Project NERA was Google’s original plan to create a closed ecosystem out of the open internet. Google documents reveal that Google’s motive was to “successfully mimic a walled garden across the open web [so] we can protect our margins.”
According to Google’s internal documents, the strategy would allow Google to extract even higher intermediation fees. A Google employee aptly described Google’s ambition for Project NERA to “capture the benefits of tightly ‘operating’ a property … without ‘owning’ the property and facing the challenges of building new consumer products.”
Google main strategy to do this was to leverage its popular browser, Chrome, to track users, by forcing them to stay logged into the browser. Google did this by logging users into the browser when they logged into any Google property such as Gmail or YouTube, and logging them out of services when they logged out of the browser.
https://mspoweruser.com/project-nera-state-attorneys-general...
https://storage.courtlistener.com/recap/gov.uscourts.nysd.56...
It goes back all the way 2010, I remember opening up chrome to try it, right clicking on a youtube channel background to attempt to download like I could on firefox just fine, and it not having the option, why would you go out of your way to restrict a user easily being able to right click and download? well, because you believe you own the web.
Never used Chrome and never will, if you use Chrome you are actively making it worse for yourself in the future once they implement enough bad policies that it becomes near impossible for almost anyone or anyone to bypass their restrictions.
--- start quote ---
So, you don't like a web platform proposal
...you may feel that your insights and experience can be valuable to help steer the platform from making what you're sure is a huge mistake. That's great!! Getting involved in web platform discussions is essential to ensure it's built for and by everyone.
...
In cases where controversial browser proposals (or lack of adoption for features folks want, which is a related, but different, subject), it's not uncommon to see issues with dozens or even hundreds of comments from presumably well-intentioned folks, trying to influence the team working on the feature to change their minds.
In the many years I've been working on the web platform, I've yet to see this work. Not even once.
--- end quote ---
"We do so love for everyone to join the discussion. It also never influences our decisions, not once"
1. Often the feedback goes completely to the wrong address. You won't stop Google from doing google things. 2. Most often the depth level at which the discussions on web standard are made will alienate most people, so instead of participating in "standards making" they turn somewhere else (1.).
The web is awesome and it got awesome because for the first 15 years of its existence it was actually very straight forward to run a web entity. But success brought ever growing companies and ever more complex interests. The discussions also vary a lot nowadays. There are still things being done to make the web more approachable but at the same time we see stuff like "Web Environment Integrity", DRM etc.
The problem is that a process that requires the public to be vigilant will eventually fail if the public cannot appoint people to be vigilant full time for them.
It also takes a lot of time. You have to read quite a few proposals, and there are literally hundreds of them, you have to participate in discussions in the GitHub issues, on the w3c mailing list, and in multiple face-to-face discussions.
Even the most technical people find this daunting because they are not paid for this (unlike the people making and promoting the specs). So even the technical people often come into an issue, voice their concerns briefly (or not-so-briefly) and are summarily dismissed.
I've seen Google engineers misrepresent and ignore any input from engineers working on Firefox and Safari, and just push their specs forward. So what chance does an outsider have?
It's a mess.
Granted, it's a better mess because so many discussions are happening in the open unlike 10-15 years ago, but it's still a mess.
As I say in the original comment > process will ... eventually fail if the public cannot appoint people to be vigilant full time for them.
And I was exactly thinking of paid full time work.
In particular, taking a fairly dry proposal like WEI, which is intended as a anti-bot/anti-cheat framework for web content, and spinning it with a shitpost title like "Google vs. the Open Web" is really not going to ingratiate you with the people who think hard about very difficult problems every day.
Is it a good proposal? Honestly I don't know. But the problems it's trying to address are real, so I'm inclined to give the benefit of the doubt to the people trying to solve them in good faith over the shitposters.
Do you really not agree that people might want the former and not the latter? You genuinely think that the standards folks are being driven by a conspiracy and not what they say they want?
There are ways to argue against WEI that don't involve the existence of enemies you have to fight. Maybe you could try them?
Wikipedia explains it similarly: In Internet culture, shitposting or trashposting is the act of using an online forum or social media page to post content that is satirical and of "aggressively, ironically, and trollishly poor quality"; it may be considered an online analog of trash talk.
Even Urban Dictionary is on board: A post of little to no sincere insightful substance. Especially a "shit"(low)-effort/quality-post with the sole purpose to confuse, provoke, entertain or otherwise evoke an unproductive reaction.
Frankly I have to assume you went out of your way (like, off the front page of a Google search even) to find a definition that you could cite just to prolong an online argument. I wonder if there's a word for that.
My definition was from urban dictionary btw, the first entry, maybe it sorts differently for different people.
Come on. I repeat: it's a complicated subject and a real problem, and a sincere but potentially flawed proposed solution. It deserves serious discussion and not a bunch of yahoo's throwing bombs about the evil corporate overlord of the week.
Most people don't have either the skill or time to do that. So we bikeshed instead.
- The WEI check will be designed with a level of simplicity that tech-savvy individuals or hackers can easily bypass. Criticisms or objections will be quieted with comments like, "You just need to initiate the browser using these 50 different settings and you're good."
- On the other hand, the WEI check will be intricate enough that an average user won't be able to circumvent it, resulting in them being obligated to view ads.
In this way, it's a win-win situation: the hackers maintain their access to an "open" web, while the vast majority (99%) of the population will navigate through a "Google" web.
Post that comment again when crypto accounts are FDIC[0] (or whatever scheme, if any, is used where you live) insured. I'm sure you'll get a different response.
[0] In ARM speak, kernel mode is EL1, hypervisor mode is EL2, and TrustZone mode is EL3. Each exception level is a higher level of privilege.
> they are losing the control on Android
What do you mean by this and what does Android have to do with trying to hold on to the net?
I work on fuchsia and can honestly say I have no idea what you're talking about. Fuchsia and android are more complimentary than they are competitive. I've noticed that when there is a lack of information, people tend to invent things that fit their narrative, but that's a really dangerous habit.
And from 9to5google.com Work on this Fuchsia project within Android — dubbed “device/google/fuchsia” — stalled in February 2021, with no public indication of how things were progressing. This week, all of the code for “device/google/fuchsia” was removed from Android, formally signaling the end of this particular avenue.
In its place, we have a lone “TODO” message, suggesting that Google may be building up something new in its place. The developer responsible for the change primarily works on Fuchsia’s “Starnix” project.
First shared in early 2021 as a proposal, Starnix is designed to make it possible for Fuchsia to “natively” run apps and libraries that were built for Linux or Android. To do this, Starnix would act to translate the low-level kernel instructions from what Linux expects to what Fuchsia’s Zircon kernel expects.
So ... custom kernel and a custom OS that will support Android applications as far as I understand ...
I participated in an experiment that tried to fingerprint without cookies over time. All browsers failed but Firefox did best (for me). So that's what I use.
However, according to my logic using Brave + Firefox simply must cause more data to be collected than using only one of the two, no?
I don't save the passwords in the browser. I keep them in Vaultwarden, installed locally on a miniserver.
I love this little bubble all of HN (or at least a vocal majority) seems to live in. Google is most definitely not collapsing anytime soon, and their products are loved by millions, if not billions, of users all over the world.
>They are losing their monopoly
No, they most definitely aren't. Brave Browser runs on top of Google's Chromium. Firefox runs on top of Google's money. Their lead in search does not seem to be going away anytime soon - there is a reason literally everyone on earth uses Google as a search engine. There is a reason literally everyone on earth uses YouTube to watch any video they want. There is a reason 70% of all phone users use Google's operating system. There is a reason Gmail is by far and away the clear leader in the personal email space.
>They have announced that they will try to block navigation if you have an ad blocker installed (for example when watching a video on YouTube).
As they rightly can. You are under no obligation to use YouTube - and if you do use it, you must pay for it, either by watching ads, or by paying for YouTube Premium.
HN can keep complaining about Google all they want, but Google is one of the few companies that has truly made the Internet the Internet. Their impact on humanity has a whole has so far most definitely been net positive, and you are under no obligation whatsoever to use their products. There is a reason they are the clear leader in the products they offer, and that is because they offer, say, a free tier (as in Gmail), or openness (as in Android).
But doesn't it logically follow that the same truth holds for any other 'strong' company, thereby rendering our perception of it (or any other company) worthless? I'm sorry you're disappointed, but I just made a logical continuation, that's all.
Well ... with this new proposal they are trying to change this, don't you think ? Yes, it is not mandatory to watch Youtube, but it should be also mandatory that Google don't collect and sell the personal data without the owner permission or scan all the emails in every Gmail account (free o paid) ... The history of Google is full of these practices and, after discovered, every time they respond "will never do it again" ...
The strategy over the years has always been the same:
1. create a necessary product and give it away "for free"
2. wait until people are used to it and consider it essential and difficult to migrate
3. close the gate and make it no longer free.
For example : Gmail for organisations (at launch free up to 100 users, then 50, then 10, then 0), Maps for websites (lower free tier now), Google Drive (lower free tier now), Youtube is next ... That these are the "best" products in the world is a subjective affirmation. They are pre-installed on devices and difficult to remove ...
They can do whatever they want with their products, of course, but trying to control the openess of the web as we know now, it is a different thing ...
To resolve this conundrum, Google as a whole cannot be said to be "for" or "against" the open web. Instead, Google's infamous internal infighting means that you can only say some parts of Google are for the open web, others are against, and sometimes one has an upper hand.
Maybe news agencies aren't much interested because this is still only an early proposal, but considering its implications I find it striking how little it seems to be talked about on the web (outside of hackernews). Rossmann seems to be the only one with a video on the topic up on Youtube. There's only a handful of Twitter results for "web environment integrity api" with next to no replies to them. When I look the keystring up on reddit the only result with a noteworthy amount of debate is not related to WEI at all. Social media is probably just on its last legs, but it doesn't seem like too many people that care are left to fight for what the web, or even general computing, used to be.
Marionette is built into Firefox so that might work, except it would require Firefox to implement this as well so it can prove itself.
Your hypothetical change of emulation tactics won't work. You're analyzing at the wrong abstraction level.
The "attestation tokens" to validate the integrity of the web browser environment would come from a 3rd-party (e.g. Google Play services).
For example... Today, hacks like youtube-dl work because implementing client-side code to "solve javascript puzzle challenges" is still inside the "world" that Google-server-to-browser-client present to each other. Same for client-side solvers for Cloudflare captchas. The "3rd-party attestation token" breaks those types of hacks.
Most interesting thing I've read all week.
Companies like Google love kicking down the ladder. You can bet that the Google crawler will have its own "attestation token" but if you want to crawl the Web with your own code you'll be SOL.
All these billion-dollar tech companies got their start thanks to open, accessible, hackable systems. Now it's all being locked down so only the big guys can play, and the rest of us have to pay a fee just to put our "apps" into their walled gardens, and if we do anything they don't like (or are just unlucky) then we get banned forever.
Let's be real here and note that while most web properties welcome Google crawlers, there are many, many other scrapers/crawlers that offer zero value to web operators while costing resources.
What we really need is to get the W3C and IETF to straight up throw out vendors who repeatedly push user hostile proposals.
Do not let anyone with employed by Google contribute a web standard. Period. And reevaluate the ones already accepted while we're at it.
https://europa.eu/youreurope/business/selling-in-eu/competit...
But yes, of course this is not just about filtering out fake clicks. The draft proposal lists a bunch of use cases, most of which have nothing to do with ads.
You're right that attribution and measuring ROI is way harder and less precise for ads sold by impression than by click. That's why they're not the common form of advertising, especially on these kinds of ad networks. But for cases where the ads are per impression, the concerns about fraud would be exactly the same. It's not about a crawler accidentally generating impressions, it's about bots deliberately doing so.
Tell my uncle, or my aunt, that "Google wants to undermine the internet of generative systems!" Whatever. Tell them "Google wants websites to be able to block any devices you might have modified, in any way, that the website owner doesn't like" and you'll get a much stronger reaction.
"Google wants to block any devices you repaired yourself" might get some traction.
Clickfaud makes the frauders money. There is a financial incentive in fooling Google's bot detection. There is a lot of bets using a faked user agent.
>Websites can refuse service unless you install their proprietary data collection agent. Websites can refuse service if you use the wrong browser
Websites can already do this.
I think we will get to a PostCapitalist future. The decisions we make in the next 7 years will likely determine whether the probable future is dystopian like Shadowrun, or utopian like Paul Mason (see his book "Postcapitalism: a Guide to Our Future").
Personally, I prefer Mason's, with his goals of:
- Rapidly reduce carbon emissions to stay below 2 °C warming by 2050 (edit: We've lost this battle, see the current 6-sigma sea ice event and recent AMOC reports - maybe we can hold it to 3 °C).
- Stabilise and socialise the global finance system.
- Prioritise information-rich technologies to deliver material prosperity and solve social challenges such as ill health and welfare dependency.
- Gear technology towards minimising necessary work, until work becomes voluntary and economic management can focus on energy and resources rather than capital and labour.
That will not be if we do not bring to heel the FAANG companies now, and prevent things like Apple's Private Access Token, Google's WEI, etc. from taking root (yanking them out of the ground where already present).
[0] https://pluralistic.net/2023/07/24/rent-to-pwn/
[1] Economists call these "rents", even though they're more general than just rent paid to borrow some real property
The usual term for that is “cyberpunk dystopia”.
EDIT: Saw a few mention two solutions to disable the automatic verification on iOS & macOS.
https://blog.cloudflare.com/how-to-enable-private-access-tok...
It isn’t perfect but we are ahead of most others (Mastodon, Matrix). We have spent TWELVE YEARS building the free, permissionless open source platform for anyone to assemble and host their own community software with all the features of Facebook/Twitter/TikTok for their own community:
https://github.com/Qbix/Platform
We are about to roll out version 2.0 — I have never done this before but I would like to invite whoever wants to learn about it or build on it, to a Zoom webinar where I will demo anything and answer any questions. Starting in Q3 this year all the webinars will take place on our own platform — no Calendly, no Zoom, no Google, just the free open Web.
Anyway, sign up here if you want. Will do it every Sunday throughout August:
https://calendly.com/qbix/qbix-2-0-platform-demo
Whether you’re a developer, a businessperson, or just want to learn about the latest technologies moving the Free Open Source Web forward, this platform can help empower you to build and engage a community around yourself and your projects.
I have lost a big part of my former trust and want for writing OSS this last few months and one thing I have learned is that if those two can't be answered with a resounding no it is a project I won't ever contemplate even though I'm neither American nor Russian.
https://www.theverge.com/2021/1/5/22215588/github-iran-sanct...
As a side note, many people on our development team that has worked with us since 2013 have spoken Russian, coming from Ukraine, Armenia, Russia, etc. Many of them continue to work together despite the war their governments are conducting.
We are for empowering people uniting communities around the world, and are pretty critical of government overreach:
https://news.ycombinator.com/item?id=35656705
https://community.qbix.com/t/transparency-in-government/234
https://qbix.com/blog/2021/01/15/open-source-communities/
If that appeals to you, the lowest hanging fruit is just joining the community and introducing yourself:
https://www.theregister.com/2023/03/21/russian_foss_contribu...
But thank you for your reply.
Why are you using Github then?
Why using calendly, zoom, and google? Well, as I said, we haven't launched Qbix Platform 2.0 to everyone worldwide yet. This is if you want to get involved pre-launch.
We dogfood our own stuff, but we also interoperate with everything else out there, such as Discourse (https://qbix.com/ecosystem for example incorporates it), Zoom, Google, Facebook, etc.
Disabling the feature on your device will make you fail attestation and thus websites requiring it will just stop working.
* CON: The problem is that the WEI server could change it's tracking faster than the browser app updates it's fakeness though. There's more money in bypassing adblockers than there is in blocking them.
* CON: If it does fake itself, when you return to the original website it can assume there's no adblocker and fail to load with the adblocker unlike now where it's usually ignored.
1. Someone could set up a server that proxies WEI required requests to regular clients. The client initiates the process, the request goes to the middleman, the middleman makes the proper WEI authorized request, gets the response, passes the response back to the client.
2. The private key could leak somehow, and so, software can forge the required signature.
I'm not holding my breath for either one. Some kind of regulation has to step in, otherwise Google puts the internet in a chokehold.
Are there stats on this?
> If I look at my own webserver logs, it’s very clear which are bot requests
Nobody ever lies on the internet?
Inadvertently at first, they were amongst the first to ever even run into some of the complexities of providing internet services at scale after all. But then the rot took hold and Google Chrome was created.
There are obviously a lot of details to work out, but pretending that this is just a power grab by Apple, Google et al. instead of an attempt to address an extremely serious problem (that HN indirectly complains about all the time, e.g. "web 2.0 sucks", "why are search results so bad now"), is just naïve.
I remember sites doing all sorts of hacks to identify and shut down IE back in the day. "Works best in Chrome/Firefox".
"The proposal calls for at least the following information in the signed attestation:
The attester's identity, for example, "Google Play".
A verdict saying whether the attester considers the device trustworthy.So a user agent string and a weak attestation?
This seems an overcomplex nothingburger.
That's not a "nothingburger" IMHO.
The goal is a verified stack - the hardware key proves you have approved hardware. The approved hardware proves you don’t have a tampered OS. The untampered OS proves you have approved binaries. The approved binaries disallow certain actions that users want such as blocking ads or downloading YouTube videos.
Is that listed in the article anywhere? Is that part of the proposal?
The proposal does however say that even if the attestation fails, that the user should be allowed to access the website.
Are you upset with the proposal, or some other proposal that you are imagining?
Same with Accelerando.
In-universe reason for that made no sense whatsoever and was nonsense. It was not even an evil organization (this would make sense) but OCR method as described was absurd and would not work.