On the other hand, real history if filled with all sorts of things being treated as a god that were much worse than "unreliable computer". For example, a lot of times it's just a human with malice.
So how bad could it really get
There were several original Star Trek episodes that explored this scenario. Not plausible. Actual.
"So how bad could it really get"
Watch Rodenberry's orginal Star Trek to get some ideas.
This isn’t anything like the trolley problem. And yes, taking actions has consequences intended or otherwise. That’s not the trolley problem either
Gene Roddenbury knew this, and it's kinda why the original Trek was so entertaining. The juxtaposition of super-technology and interpersonal conflict was a lot more novel in the 60s than it is in a post-internet world, and therefore used to be easier to understand as a literary device. To a modern audience, a Tricorder is indistinguishable from an iPhone; the fancy "hailing channel" is indistinct from Skype or Facetime.
https://www.telegram.com/story/news/local/north/2010/06/22/8...
"Ms. Sackett, with the aid of film clips, said that "The Return of the Archons," from the original series, was a good example of how Mr. Roddenberry employed elements of humanism in his works.
In that episode, a planet's population follows, in a zombie-like manner, a mysterious cult-like leader, who allows no divergent viewpoints.
The society absorbs individuals into its collective body and the world is free of hate, conflict and crime but creativity, freedom and individualism are stifled.
Ms. Sackett said that "Archons," like other Star Trek storylines, warns how people can be controlled by religion. In the end, the viewer discovers the cult leader is actually a computer."
"[N]o divergent viewpoints" sounds like Stackoverflow and forums run by software developers in general. The behaviour of "developers" can be extremelly cult-like.
Creativity, i.e., new work that is not comprised of a recombination of old work, does not seem compatible with "AI". The later relies on patterns found in old work.
"As bad as it can get" is an AI that, either by accident or due to malign influence, takes "I Have No Mouth, and I Must Scream" as a guide book.
Actually, I take that back, it would be what happens in the hells in Surface Detail.
I don't know. How about we ask some of the peoples who have been destroyed on the word of a single infallible malicious leader.
Oh wait, we can't. They're dead.
Any other questions?
How does doing it with a computer add anything?
Say what you will about a human, but unless you're a religious zealot or blind you generally don't believe the leader to be infallible. But through the magic of silicon you can shut people up more effectively.
This makes computers an accelerator of the problem, and therefore warrants caution any time their output may be relied upon for life and death decisions.