The asymmetry of nudges
lcamtuf.substack.com
lcamtuf.substack.com
What we need are external checks and balances. These can come in many forms from market competition, to government regulation, to watchdog groups. Putting pressure on individuals to change massively powerful systems from within is a fools errand.
But I have room enough in my heart to also hate the individual engineers making boatloads of money actively worsening the world around them.
There's a tautological interpretation of what you're saying which, since it's tautology, is always true. It relies on "corporate benefit" being whatever the responsible parties in the corporation decide it is.
But in the more usual "fiduciary duty" sense where "benefit" means direct impact on the bottom line, then no, this isn't true. Corporations are run by people, and not only can those people decide to forgo some profit in order to do the right thing, this actually happens from time to time. A ratchet prevents movement except in the designated direction, or it isn't a ratchet: so a ratchet is a bad metaphor here.
But, as the article also argues, there's a decided asymmetry to the flow here. Proposals which benefit the bottom line are easy to make, and easy to achieve buy-in on. Ones which have the opposite effect cost reputational capital, they're risky. So even a well-meaning corporation which is entirely owned and operated by people with a firm commitment to a vision of corporate benefit which isn't blindly determined by the bottom line, will miss opportunities to fulfill that vision which would negatively affect that bottom line, for structural reasons.
That is an unsatisfying conclusion as the general structure of Google is unlikely to ever change, but it does seem correct to me.
The real structural problem is that the needs of the shareholders and by some extension the needs of the high level executives and managers at Google are simply not aligned with the needs of the users. This is why the “nudges” inch along in a direction which is often at odds with the needs of the users.
The solution to this broad class of structural problem in our economy, as argued by economists like Richard Wolff, is to build our economy out of firms which are largely cooperative in structure, where the workers and members of the co-op are representative of the users of the product or service. For example if your local water company is a co-op of users, with cooperative decision making power, the co-op isn’t going to vote to raise water rates unnecessarily against their own users.
A middle ground in many cases is unions. So if anything this article is unintentionally making a case for a tech workers union at Google. This would change the structure at Google in the most significant way currently possible under today’s legal system.
I think the idea that engineers should take more responsibility is a noble one, but it’s not the real problem here. The problem is the structure of the firm.
And who gets to define what “right” is?
However I also see people respond to me when I talk about co-ops who somehow think I want to be Joseph Stalin, where in the imagined case they are skewering me for wanting all the decision making power. But that is a fantasy as the whole point of a co-op is to have more diffuse decision making power than under traditional orgs!
My interpretation of the article is different.
I think the author is merely drawing attention to a force that pushes businesses in a particular direction, without proposing any specific solution.
The article doesn't say he had an alternative design or vision he wishes he had pushed harder, or anything like that. In fact he's full of praise for Manifest V3, which he considers the most elegant technical solution to a real problem, which he considered an indefensible security and privacy risk.
“And we — the well-meaning engineers — shoulder much of the blame.”
It’s more complicated than that and there’s no way to discuss things if someone takes that stance.
The article wasn’t trying to convince you the changes was good or bad.
People can do what they see as the right thing (limiting plugins) for the right reason (they had WAY too much access to data), and despite no evil being intended or agenda to protect the ad business existing, people come to that conclusion.
This facts are just all more complicated than the common talking points you see. That’s it seems to be trying to show.
The status quo alternative is elite ownership of the utilities and manufacturing facilities, where those individuals receive a large share of the profits which they can use to insulate themselves from environmental pollution or cost of living concerns.
The fact that co-ops aren’t absolutely perfect in all cases is not strictly a knock against them. The status quo is particularly troubled.
The article seems to be trying to argue that company leadership are not the ones responsible for the "evil" things that companies do. But this:
> If you’re an engineer at Google, Facebook, Apple, or Microsoft, it’s always easier to propose architectural changes that don’t hurt the bottom line, or perhaps bolster it by accident. Conversely, if your proposal stands to wipe out a good chunk of revenue, you either self-censor and don’t bring it up — or you end up getting sucked into endless, futile arguments.
strongly implies that company leadership are indeed the ones responsible.
Like, I think what the article is trying to say is that, Manifest V3 was designed due to real-world privacy concerns, not for profit motives. It just happened to get the right amount of support and buy-in from leadership due to being something that -also- aided profit motives.
In other words, when a company leader has a variety of possible projects to invest in, she will naturally tend to invest in the ones with a long-term profit motive for the company. This also necessarily means -not- investing in other, potentially good and helpful and consumer-positive projects, that simply aren't as promising from a profit perspective. This phenomenon is what the article calls the "asymmetry of nudges".
But I guess what I'm failing to grasp is how this means it was the engineers' doing and not leadership. Yes, the engineers came up with the idea. But in this scenario, it seems like the engineers were the ones who were well-meaning, and just doing their jobs. Whereas leadership were the ones chasing dollar signs at all costs. This is precisely in alignment with what most people posit when they say that big corporations are evil, no?
“It is difficult to get a man to understand something, when his salary depends on his not understanding it”
Even the most well intentioned engineers aren’t going to propose something that will dramatically impact the company bottom line. They are “nudged” (and the roadmap prioritized accordingly) to fall in line with the best interest of the company.
> And we — the well-meaning engineers — shoulder much of the blame.
Which doesn't seem to align with that understanding.
You want ethics to be a factor - you must introduce regulation so it becomes a non-zero weight in the solution space search engineers do.
Someone in leadership could literally flick their wrist 3 times and MV3 is dead.
This doesn't sound like an equal responsibility type of situation to me.
The engineers would have to stage some kind of protest and/or quit their jobs to change things. Whereas all the leaders have to do is just stop choosing to invest in harmful projects.
While I do see how it's one of those situations where, the system is set up to incentivize certain decisions. I get that. But that doesn't change the fact that the leaders are the ones making the decisions.
Maybe because the article doesn't claim that. (Shouldering some of the blame is not the same as what I quoted) It just presents different incentives that push decisions over time. The incentives that end up putting the engineers to do something in the end don't mean it's an engineers' fault.
There's no point choosing one specific group to point fingers at, if we can instead learn more about the system and if we have the power, try pushing it slightly in a better direction.
From the perspective of a typical HN reader, Google and Mozilla have turned into Internet nanny states with Fisher-Price browsers. How far can they go in the name of "safety" before it's too far?
Not to mention the problem the article highlights: their motives aren't pure. The more control they give themselves, and the more inconvenient third parties they marginalize, the more money they stand to make.
Also, it's not a perfect A or B between flexibility and security. They could require extensions to be more open and inspectable so users could catch bad behavior. They could better police the extension store to catch malware faster. They could add more layers of warnings and permissions dialogs to prevent accidental compromise.
At any rate, whether due to incompetence or malice, the situation is not as one-sided as Google pretends it is.
nope. "legitimate" extension authors get compromised - either their creds or they sell the extension to some other less reputable group.
For example, Boeing moving its headquarter, so the decision makers are far away from the reality on the ground. This pattern is visible in less extreme ways in most companies. CxO's are typically on another floor than the other people.
The idea is clear: They don't want to know what happens in reality. They want to be able to deny anything, while nudging everyone in the right direction.
TFA didn’t do this. They posit a kind of passive malevolence, where things that hurt the bottom line are forbidden, but everything else is fine
As in, a content filter extension (or anything that interacts with a content filter) is run in a WASM sandbox without any access to the network or underlying system? It’s hermetically sealed from the rest of the extension, that might well need to make external requests to function.
The v3 fixes this by instead only letting bad blocker submit things they want blocking, and never letting them see that page. It’s not perfect by any means, but it is much more secure.
For example, iOS runs content filters in a special isolated process with no persistence or access to the outside world.
With this model, the content filter requests elements to be removed and the browser does the actual removal. As such, the scope of modifications can be reduced whilst keeping arbitrary and perhaps complex filtering logic.
“Send it somewhere else” being the important part.
The interface Chrome went with was a declarative list of filters that Chrome will use to perform the actual filtering, but the declarative interface isn’t great and wasn’t well received.
My point is: there is an interface that is non-declarative and sandboxed, whilst allowing Chrome to perform the actual filtering.
nope - the way ublock etc work is arbitrary code exec and arbitrary access to the urls and pages.
the new extension model that everyone hates is extreme sandboxing, but not letting adblockers do that.
In other words, the problem is poor sandboxing of extensions.
If your company has an incentive to make products hard to fix by the lay person over time your company will make decisions that leads to precisely that, even if most of the individuals involved by themselves had a principled stance towards the quality of their designs, products and repairability.
That means the only reliable way I as a customer can trust a company means this for real, is if something within their structure disincentives selling out their good reputation for short term gains by creating shittier less repairable products.
The problem is that in capitalism most organizations are structured with incentives in mind that don't care about long term effects on the environment, society or even the company itself.
This does not follow from the rest of the article at all. I'll begin by acknowledging the concept of the "asymmetric nudge" as a useful thought. It does somehow explain and ground a feeling of engineers within large corporate structures, where somehow all of your good ideas turn user hostile. The author fails to sufficiently answer the followup question though. Why are the nudges asymmetric, and who holds responsibility for that?
This is where the "sociopathic" executive comes in. The executive does not make technical decisions. Instead they make human decisions, like what projects to fund, what form of communication to accept, and what sorts of arguments to listen to.
The power of the executive is not to censor designs, it's to instill the values into you that steers your self-censorship.
Welcome to the modern executive 101. If you are ever directly culpable, you aren't nudging well enough. You try to structure things around the peons to make them do that thing you want; but in a way responsibility never bubbles back to you.
As a "peon", your moral job is to make that impossible by not tolerating hand waves, and pinning execs down into giving a clear, traceable, accountable order. Even if it makes them uncomfortable. If they aren't made uncomfortable, you aren't doing it hard enough.
> One of these had to give, and Manifest V3 was the most elegant technical approach. Far from being the brainchild of a sociopathic executive, its architecture was devised by well-meaning engineers on the Chrome team.
The Chrome team has some very competent engineers. lcamtuf is a well-respected security engineer. I would expect such a group, trying to solve a problem of poorly behaved extensions, to develop a nice privacy-respecting API to block requests.
For example, there could be a way for an extension to run a portion of itself in a sandbox, such that the sandbox could inspect a request, decide whether to allow it, and output only an indication of whether to allow it. No further outgoing communication, including to the rest of the extension, would be allowed.
But instead we got Manifest V3, and I simply don't believe it's a meaningful privacy improvement. Read the docs: https://developer.chrome.com/docs/extensions/reference/api/w...
> Note: As of Manifest V3, the "webRequestBlocking" permission is no longer available for most extensions. Consider "declarativeNetRequest", which enables use the declarativeNetRequest API. Aside from "webRequestBlocking", the webRequest API is unchanged and available for normal use.
Did well-meaning engineers on the Chrome team really come up with a security improvement in which extensions can read request and response headers but not block the requests? I'd love to see an explanation, but to me it seems that the security "improvement" is pretty narrowly tailored to prevent ad-blocking without meaningfully improving privacy.
you'd think with their legions of competent engineers they'd be able to come up with some way of defeating this attack
but that would hurt the business over the blunt MV3 approach, and you're not going to get promoted for that...
This is a really awkward attack for a couple reasons. In general, a malicious extension may have no way to tell whether a request was blocked — the origin if the request doesn’t belong to the extension authors, and the portion of the extension outside the sandbox won’t be told which requests were allowed. And, if too many requests are blocked apparently at random, the user may well notice.
It’s surely possible to sneak out some data, slowly, over a noisy channel, but it doesn’t sound straightforward.
Compare to actual manifest V3, where exfiltrating the keys to the kingdom appears to be entire trivial as long as the extension doesn’t try to block ads.
> In fiction and in journalism, the fault almost always lies with the executives
> we — the well-meaning engineers — shoulder the blame
This is a weird take to be honest. Company culture is the responsibility of the executives, and however we put it, ultimately the blame lies on them.
Is the hell paved with good intentions ? yes, surely, and there's a need to be critical of the impact of one's work. We could fault people for not taking a step back to look at it from a distance.
But the reward ("nudges") system the article is focusing on isn't that, it's incentives put in place by the company. Who set up these incentives should get the blame when shit hits the fan.