Hacker News Reader

Computers are fine but total computerisation of human society may not be so. That's actually what the article should have explored in my opinion. Computers are neither ruthless nor kind, its controllers, humans, are. So the question then becomes, who controls them and how sophisticated they (the computers) become, and how much they infiltrate our world. A simple thought... should everything that can theoretically be controlled by computer algorithms, made so? Should computers use ML for as many as tasks as possible, even though that can lead to wrong decisions on occasion?

Institutions made of people have limited reach. Human resources are finite, there's a physical limit on the scale of what they can do. A computerized society is limitless. A state might want to spy on everybody but it's impossible to do so due to limited manpower. Computers remove those limits and allow states to implement global surveillance.

Computerized societies are still limited by their resources. We are all seeing now the effects of a society that thinks it can print money without ruining its money supply.

Computers still scale much faster than humans.

Whether computers count as ruthless depends on fine details of your definition of ruthless.

If you mean the mental/emotional state necessary for a human to act without regard for consequences of their actions to other people, then computers are not ruthless.

If you just mean acting without consideration of how their their actions will affect people then computers fit this definition of ruthless perfectly by virtue of not being able to consider the consequences of their actions on other people, because we don’t know how to program them to consider how their actions will affect other people.

> If you mean the mental/emotional state necessary for a human to act without regard for consequences of their actions to other people, then computers are not ruthless.

I don't understand, what am I missing?

I've never seen a computer have any regard for the consequences of its decisions. Computers are thus completely ruthless.

I think the GP points to the fact that given the inability of computers to evaluate the consequences of its programming, ruthlessness is a concept that cannot apply to them because by definition involves disregard about those consequences, not ignorance.

edit: typo

I think there is a conflation of malicious and ruthlessness in this thread, showing no pity or compassion (def of ruthless) does apply to a computer because they can do neither. However, being actively malicious and disregarding consequences probably only applies to humans controlling them because the computer is ignorant of those consequences as you say.

> because we don’t know how to program them to consider how their actions will affect other people.

We don't have to. For example, a ruthless train door could be made less ruthless, by having sensors that respond to someone who is nanoseconds late, to open the door, just that once, just like it's elevator door brethren, but only once.

The problem is that "inherently opportunistic people" will take advantage of the machine kindness and take an entire train full of people hostage.

The oppressive train door is a dictator everyone loves, maybe if they were also razor sharp, people would give them the respect they deserve. phwump. guillotrains. on time, every time.

But doing so isn’t the default state for computers. Someone needs to think about that, and put effort into building that. Making it so that some concerned citizen that wants to make the system less ruthless, also demands the instrumentation of the system to allow for that, cooperation between all parties to facilitate such changes - it doesn’t happen by default. So while it’s possible to make a system not behave ruthlessly with enough effort, it is default ruthless, for better or worse.

The alternative mechanisms for decision-making also make wrong decisions on occasion, especially when that alternative is a human.

> So the question then becomes, who controls them and how sophisticated they (the computers) become...

There's also unexplored (by article) concept of inadvertent or emergent control.

E.g. Dept A encodes Rule 1 and Dept B independently encodes Rule 2, yet applied together Rule 1 and 2 produce an unexpected outcome. In which case, neither Dept A or Dept B (the ostensible controllers) could be said to actually control the outcome.

infiltrate? They're not spies. They're savants at best.

There is also the question of increasing human-machine integration, eventually resulting in a direct read-write interface to the brain (see Neuralink, OpenWater, and VALVE's research and goals), in the context of proprietary software under corporate and government control.

I agree.

I think the oppressiveness will come from software. Who writes it, and what values does the software reflect? Does it reflect yours?

The governance structure wants to code their rules into a machine that will execute those rules without being able to apply human discernment. Humans are meant to be out of the loop - their rules and diktats are meant to be all we need. Ai + robotics has brought this possibility onto the horizon.

The issue really then, is in people's acceptance that someone can write meaningful rules for them to follow. Hence we see jobs such as AI ethicist. As if they can do anything other than justify the immoral decisions corporations/government make.

The collective that still believe government has their best interests at heart and are unable to refuse following the worst of us, will drive us into technological tyranny.

This is exactly why free software was invented, which respects its users: https://www.fsf.org/blogs/rms/why-free-software-is-more-impo....

It's odd for an article like this to make absolutely no mention of Free Software, but it's not really using oppressive in the sense the Free Software movement uses it.

The point of the article boils down to something like this:

Pre-AI computers are very literal and show no judgement or discernment in their rule-following. They are inevitably used to implement complex systems against which users have little recourse, and to implement complex systems whose behaviours and failures can be blamed on computers rather than on the people responsible. These fundamental problems will persist no matter the progress made in AI.

Free Software is intended to stop the user being oppressed by their own computers. The article's point is about other people's computers being used to serve their interests over yours, in ways that may be detrimental to the user compared to a more 'manual' system.

Consider the train door example from the article. Suppose that code were Free Software. This might benefit whoever owns and runs the train, but wouldn't benefit the user (of the train) at all.

> The article's point is about other people's computers being used to serve their interests over yours

Partly, but this example was also used as well:

> Though perhaps unlikely, my hyper-awareness of the sinister ruthlessness of machines in the ordinary things around us led me immediately to note the pathological outcome: Someone stuck on a desert island (or equivalent) who starves to death, because the only food they have is expired Juicero juice bags and a machine which refuses to serve them. While in Juicero's case this might seem like a comical example, I fully believe it's only a matter of time until something like this happens. The deliberate utilisation of machine-assisted ruthlessness to malevolent ends becomes more and more common with every passing day, as companies realise they can escape from accountability by hiding behind the facelessness of the machine, which disempowers the individual to fight or object to it. The now memetic expression, “computer says no”, pithily expresses this phenomenon.

…which is precisely the usual free software issue. The Juicero is a computer you own, but that does not serve your interests.

Yes, good point.

> Computers are neutral technology.

Don't make me tap the sign: Technology. Is. Not. Neutral.

A shame so few technophiles have read Jacques Ellul[1]. Ellul's La Technique (and his later "Propaganda") should be mandatory reading for anyone thinking about systems. Be careful though, this thinker/philosopher has verbalized a problem that made Ted Kaczinsky conclude the only solution is for us as a species to retreat into the forests and vanish into the trees. Read Ellul at your own risk.

About the rest I agree, but this is complaining about a system that works _exactly_ as intended. Read Ellul!

[1] "La Technique" (The technological society) https://archive.org/details/technologicalsoc00ellu

A 500+ page book is beyond most peoples tree time to explore from a comments faith alone so here's a summary I found (the quality of which is unknown) since I'm always interested to some degree in new perspectives (https://www.supersummary.com/the-technological-society/summa...):

While Ellul separates technique from the individual machines that contribute to its reign, he maintains that machines represent an ideal state for the forces that govern technique. Humans, on the other hand, are not ideal at all. We lack the efficiency and predictability of machines. Therefore, Ellul says, technique is the social adaptation that exists in transforming the messy lives of humans so that they better fit into a world controlled by machinery. Further, in a somewhat terrifying turn of phrase, he adds that technique is "the consciousness of the mechanized world."

So humans adapt to technology and as technology becomes more efficient we just become cogs of technology is the core principle here and I can see how this relationship could be thought to arise however I completely disagree with it. Technology does enable efficiency at a fundamental level but it's not that humans become slaves to the efficiency or technique, it's that some humans chose to force other humans into increasingly more efficient means.

There is some feedback relation here in that in a global competitive environment as leaders around the world continue to compete in various forms they will inherently drive their populations into deeper efficiency I believe but the rise to all of this is because of the global competitive environment. It's largely social constructs and to some degree, finite resources that drive these behaviors and lead to efficiency focus. More than anything it's greed and hunger for power that drives this nonsense.

Most humans recognize when their lives become dominated by some silly efficiency and don't care to follow suit. Their want autonomy, to own their life and own their choices. They're instead forced into "the technique" so to speak as a secondary effect. The underlying effect is what needs to be rutted out, technology isn't the fundamental issue here, it's a set of human behaviors and institutions that exist and enable them.

so this is a review of a review + your conclusion that you disagree? a shame you didn't read it because it is less about machines but about systems (la technique) and systems thinking. There is a whole chapter on Propaganda which he expands on in a later book.

>This oppressiveness is not inherent to computers, it's inherent to the people and institutions making use of them. The examples in the article say it all: government bureaucracies, beverage dispensers.

Well, to build things at the modern computer industry scale, you need government bureaucracies, huge corporations, tons of discipline, from the assembly line to the cobalt mines, and enterprises and bureaucracies with a use for it, 99% to streamline control, control more effectively and wider, and penny pinch customers even better.

Now, give surveillance, instant indexed cataloguing, 24/7 tracking, remote controllable money and id access and more, to the same governments and corporations, and that's just going to point to more control.

The best thing about this topic I've read is an article by Melvin Kranzberg as "Kranzberg's laws" https://doi.org/10.2307/3105385 .

There are six laws.

1) Technology is neither good nor bad; nor is it neutral.

2) Invention is the mother of necessity.

3) Technology comes in packages, big and small.

4) Although technology might be a prime element in many public issues, nontechnical factors take precedence in technology-policy decisions.

5) All history is relevant, but the history of technology is the most relevant.

6) Technology is a very human activity-and so is the history of technology.

tools are not neutral: they favor that which they make easier

Computers make a lot of things easier, both oppressive and subversive, just like any other highly versatile tool: hammers, screwdrivers, knives, syringes, duct tape and WD-40.

Tools define a relationship with the entity being tooled, so you can learn a lot about the goals and values of the transactions by looking at the kinds of relationships that are/aren't implied.

You can't feed people with an AR-15.

The metaphors that define computing are all about abstraction, automation without human input, predictability, and control.

Abstraction tends to lossy and context-insensitive, which is why an algorithm will say "no" to a loan request based on a credit score that ignores personal context, where a human loan manager who knows the people involved might say yes - for good and valid reasons.

Computing seems to be one of the most successful products of modernism, which values speedy predictable repetition, lack of decoration, and anonymised mechanisation.

It's possible to imagine computing in other value systems. But it's revealingly hard to do without losing the essence.

All of which is a complicated way of agreeing - computing is not neutral, and very probably can't be.

It's the first automated value evangelising system in history. And that's not a comfortable thought.

You can feed people bullets with an AR-15, but that usually doesn't end up being a life extension measure :(

> Abstraction tends to lossy and context-insensitive, which is why an algorithm will say "no" to a loan request based on a credit score that ignores personal context, where a human loan manager who knows the people involved might say yes - for good and valid reasons.

A human loan manager, given only a credit score, will approve the loan based on only the credit score. The problem isn't human vs machine, it's context-awareness and context-sensitivity vs a lack of one or both.

All large systems, human or machine, tend to be implemented using abstractions. Layers of context-stripping abstractions were a part of buerocracy far before machines took it over. If a loan request form doesn't have an "if I don't get this loan my family will literally starve to death" box that you can check, the evaluator will not know about that context couldn't possibly act based on it. The form can be digital or paper and the evaluator can be a program or a human - it doesn't change a thing.

Similarly, context-sensitivity (the willingness to let the context sway your decision) was strictly limited even back in the day when decision makers were close enough to the source to be aware of the context. A bank clerk had to stick to the rules and couldn't just decide to give a loan to someone that didn't qualify under them. If they did, they'd be risking losing their job or worse, so they didn't. This discretion was only given to the people in charge - and that hasn't changed with computers. A person in a high enough position can override what the computer says, just as easily as they can override a lower clerk's decision.

Yes, making machines context-aware takes some effort, and making them context-sensitive takes even more effort. But even if machines magically came with both, their operators would do the best to get rod of them, just like they did with their employees. From their point of view, empathy is inefficiency. And that's the real problem.

I get your point, but consider that you can hunt with an AR. It’s pretty popular for women hunters as a deer rifle (which I found surprising).

Okay, but consider that I might be better off with a machine that approves or disapproves based on credit history and credit score than a banker who won't lend to me unless I get my husband's signature because he doesn't think women should be making choices about money without their husband's approval. (But who wouldn't even blink at loaning money to a man without his wife's signature.)

As a thought experiment why not go to the extreme and see if the "just a tool, it's about what people make with it"-point would hold there.

"Gas chambers are just a tool"

But a tool that has very few uses besides large scale genocide. So of course it would be ridiculous to argue the gaschambers are just a tool, because they are (A) the result of an engineering project that tried to kill as many people as possible in a short time, without using valuable war resources like amunition, so the "tool" cannot be isolated from the context of it's creation, and (B) there is one thing gas chambers are good at and it is mechanizing genocide. One could potentially come up with other things they could be used for, but there is just the one thing theyr really excell at.

Computers have more uses, but they are "not just a tool" either, because in reality nothing is.

Gas chambers are widely used for sterilising materials going into clean rooms for compounding of medicines. While this does meet the definition of “one more thing”, it is rather an important one.

With showerheads?

Not that I've seen. Nothing about the phrase "gas chamber" to anyone knowledgeable about pharmaceutical equipment suggests shower heads though - it's a simple case of you attempting to move the goalposts to support a bad argument.

> We no longer define our choices, we choose among the options they provide for us. Freely programmable computers are actually quite powerful and subversive which is why there are constant attempts to control them [...] Computing freedom is therefore against their agendas

Yes, exactly.

Spot on! The matter at hand here isn't "human vs machine" but "human vs machine vs human", meaning the oppression of human by human via the machine

> This oppressiveness is not inherent to computers

Absolutely. I vehemently disagree with this essay. One of the reasons I am a passionate technology critic and sceptic is that I love technology and feel disgusted at the way it's co-opted and abused.

I am always vigilant not to become jaded or lose sight of the immense positive possibilities for digital technologies. But I sincerely believe that before that can happen and we can move into the true golden era of technological humanism, Big Tech, surveillance culture and over-bearing governments must be eliminated.

> it's inherent to the people and institutions making use of them.

I disagree. It resides within them and is amplified by technologies. "Inherent" implies that it is an indelible, fundamental feature. Not at all. Humans, as individuals and institutions can do better. We're just seeing amplification and facilitation of the same old human weaknesses, laziness, greed, stupidity, fear - go pick up some Aristotle, or any religious text for a comprehensive list of moral failings.

> Computers are neutral technology.

Not quite. All technologies carry values in their design. An AK47 and a baby incubator carry different values, one is a for taking life and one is for preserving it. At a more subtle granularity computers carry values with them. I recommend Edwin Black's "IBM and the Holocaust" for some insights here. A Google Pixel carries different values than a Raspberry Pi - though to most eyes the differences are invisible.

> It's just that we are increasingly losing control of them. We're no longer the ones making the rules, the corporations and governments are.

Yes, computers have enabled tyranny. That's hardly debatable now. I call it techno-fascism. Other people use different terms. It's there in everything from Facebook to "cashless commerce".

> The computers belong to them now

So long as computers are controlled by software, which is mutable, the more complete and devastating will be the reversal when we take them back. There is always poetry and irony in ultimate justice. One way of looking at it is the tyrants are simply building the apparatus to enable the greatest and most audacious bloodless revolution in the history of mankind. The future of freedom depends on "hackers".

Sell them more rope.

The more neutral/academic term (instead of techno-fascism) is "algorithmic control"

Plenty of great reading: https://scholar.google.com/scholar?hl=en&as_sdt=0%2C44&q=alg...

Or cybernetic governance.

Although I am not sure that captures the problem of technofascism.

Algorithmic/Cybernetic Governance is not malicious. It's just stupid. It's a mistake made by well meaning but misguided and poorly read people who have a vicariously omnipotent fantasy that machines can run things. Ultimately it's their intent to step back and bask in an autonomous land of milk, honey, magical plenitude and obedience.

Technofascism has a malicious and power hungry side. For that lot, computers are simple weapons of enslavement.

I also think the Cyberneticians are more honest and open, They really believe in their ambitious but naive vision. The technofascists are duplicitous. They know people won't buy their control fetish, so spend a lot more on presentation, sugar frosting, and outright lies about the "benefits of digital technology", and their own intentions.