We were supposed to break; limits, barriers, status-quos, ossified ideas... Instead we broke; treasured social norms, privacy, mutual respect and dignity. There's a difference between benevolent innovation and reckless iconoclasm. I think it started the day Peter Thiel gave money to Mark Zuckerberg.
Their barbarous "barriers", "status quo", "ossified ideas"
vs.
Our blessed "privacy", "treasured social norms", "dignity"
https://en.wikipedia.org/wiki/Meta_Platforms#History
>> '"On May 2, 2014, Zuckerberg announced that the company would be changing its internal motto from "Move fast and break things" to "Move fast with stable infrastructure".[40][41] The earlier motto had been described as Zuckerberg's "prime directive to his developers and team" in a 2009 interview in Business Insider, in which he also said, "Unless you are breaking stuff, you are not moving fast enough."[42]"'
ofc, with these stale game references around, I soon got reports of broken things: targeting was broken, pvp was broken, fade out animations were broken
A few hours later these issues were resolved. The players are used to these things happening sometimes. It's fine since the stakes are low. It's just a game after all. & being free, the active playerbase understands that they're QA
Facebook was famously unit-test sparse, for example.
> The earlier motto had been described as Zuckerberg's "prime directive to his developers and team" in a 2009 interview in Business Insider, in which he also said, "Unless you are breaking stuff, you are not moving fast enough."
It’s about growth at all costs and then once Facebook got big enough they had to balance growth against other factors (+ the things people were doing that were causing breakages weren’t actually helping to grow).
Maybe true. But then if it's just about development it's a rather mundane old chestnut about reckless engineering versus good software engineering etc. Granted, that's a different discussion and we can see the tide turning now in terms of regulation and mandated software quality.
Sure, the Post-Office/Fujitsu scandal, Boeing etc, show how bad software actually ruins lives, but for the most-part the externality imposed by the reckless software engineer is measured in "hours of minor inconvenience".
That said.. I wonder if you did a ballpark calculation of how much harm lies behind the Google Graveyard [0], whether the cost of what is broken outweighs the benefits of it ever having been made?
One person’s recklessness is another person’s calculated risk. The consequences of FB engineering mistakes are minimal in both impact to customers and FB’s business. As FB scaled, the impact to individual people is still largely minimal (perhaps even beneficial) but the impact to their own business is larger and same for their customers if their ads aren’t getting eyeballs. So they shifted as big companies do. It’s kind of the best case of thoughtful risk taking - we’re rolling out a new system and we don’t know what could go wrong at scale and we put in monitoring of what we think we need. If there’s problems we’ll catch it with our monitoring/alerting and rollback or fix. You see the outages but not 99% of changes that go in without anything going wrong which lets the business resolve issues quickly and cheaply.
As for Boeing and Fujistsu, I’d say those are very different situations and aren’t an engineering problem nor do they indicate a move fast and break things mentality. As with many things like that, the engineering mistakes are a small detail within the overall larger picture of corruption. Boeing wanted to escape being classified as a new aircraft and met a perfect storm of skimping on hardware and corrupting the FAA through regulatory capture. I don’t fully understand Boeing’s role with the recent failures as a subcontractor is involved, but my hunch is that they’re nominally responsible for that subcontractor anyways. Same goes for Fujitsu - bad SW combined with an overly aggressive prosecution mandate and then cover ups around having made mistakes based on the assumption that the SW was correct rather than assuming new SW that hadn’t run anywhere before may contain bugs (not really sure whether Fujitsu hid the bugs or if politicians did or what happened but certainly the Post Office officials hid the reports of the auditors that found bugs in the sw and continued with prosecutions anyway).
Btw in engineering classes, all the large scale failures we were taught about involved some level of corruption or chain of mistakes. A contractor not conforming to the engineering specs to save on costs (valid optimization but should be done extra carefully), overlooking some kind of physical modeling that wasn’t considered industry standard yet, kickbacks, etc.
The difference then, morally, is whether the harms are sudden and catastrophic or accumulating, ongoing, repairable and so on. And what action is taken.
There's a lot about FB you say that I cannot agree with. I think Zuckerberg as a person was and remains naive. To be fair I don't think he ever could have foreseen/calculated the societal impact of social media. But as a company I think FB understood exactly what was happening and had hired minds politically and sociologically smart enough to see the unfolding "catastrophe" (Roger McNamee's words) - but they chose to cover it up and steer the course anyway.
That's the kind of recklessness I am talking about. That's not like Y2K or Mariner-I or any of those very costly outcome could have been prevented by a more thoughtful singular decision early in development.
You now have raised a totally separate issue about the overall strategy and business development of the company which you’d be right about - if it were required to have a PE license to run an engineering company, Zuckerberg would have to have had his PE license revoked and any PEs complicit in what they did with tuning for addictiveness should similarly be punished. But the lack of regulation in any engineering projects that don’t deal directly with human safety and how businesses are allowed to run is a political problem.
But on the issue I'm really concerned about, do you think "tuning for addictiveness" on a scale of about 3 billion users goes beyond mere recklessness, and what do we do about this "political problem" that such enormous diffuse harms are somehow not considered matters of "human safety" in engineering circles?
Is it time we formalised some broader harms?
I've been deeply critical of the appalling behaviour of the Post Office and Fujitsu in the Horizon scandal but there's a world of difference between this and the impact of Facebook in 2009. One had a foreseeable and foreseen impact on people's lives. The other was a social network competing with MySpace and looking for a way to monetise its popularity.
You're absolutely right there,
Frances Haugen's leaked internal communications showed incontrovertibly that internal Facebook research had long known teen girls had increased suicidal thoughts and obtained eating disorders. Facebook and Instagram products exploited teens with manipulative algorithms designed to amplify their insecurities, and that was documented. Yet they consistently chose to maximise growth rather than implement safeguards, and to actively bury the truth that their product caused deaths [0]. Similarly the Post Office had mountains of evidence that its software was ruining lives yet engaged in a protracted, active cover-up [1].
So, very similar.
But what's the "world of difference"?
> looking for a way to monetise its popularity.
That's a defence? You know what, that makes it worse. The Post Office were acting out of fear, whereas Facebook acted out of vanity and greed. The Post Office wanted to hide what had happened, whereas Facebook wanted to cloak ongoing misdeeds in order to continue. Simply despicable.
Way I see it - Facebook come out looking much, much worse.
[0] https://www.npr.org/2021/10/05/1043377310/facebook-whistlebl...