If anything happens in the society there's
always a person who made the decision
People start walking on it. In all kinds of directions.
Slowly paths through the grass start to emerge.
Those paths attract more people to walk on those.
The stronger the path through the grass, the more people use it.
Finally, there is one clearly visible path through the grass everyone uses.
Who made the decision?
The first person who walked the final path. Everyone who agreed with that person and walked the same path compounded his decision. Everyone who crossed the field should have thought about what they were doing at the time.
Far too many people, and especially developers, are way more focused on absolving themselves from responsibility for anything bad than they are willing to put the thought in at the beginning, and that's the root cause of a lot of problems.
I'm aware of how they work. I'm arguing that someone who builds a system is responsible for it even if they don't know how it works. Ignorance, even in the face of a system so complex that no person could ever understand the underlying causes of what it does, is not an excuse. No one should be able to hide behind complexity.
Developers must either build in protections against their systems going wrong or they shouldn't deploy them.
Every step of the development ladder is fraught with possibilities for error and catastrophe. To quote James Goldman:
"It is too simple. Life, if it's like anything at all, is like an avalanche. To blame the little ball of snow that starts it all, to say it is the cause, is just as true as it is meaningless."
Yes.
That wouldn't even be a change to the current system. That's how it works now. If you take on the role of CEO and it turns out that years earlier the company did something terrible you will be expected to resign. It's one of the reasons they demand so much money.
Contrary to popular belief, the vast majority of career CEOs are good hardworking people. Like any high profile position, the outliers skew perception for everyone.
Other systems may or may not be created in a similar fashion.
(Source: Have walked through fields.)
Desire paths: https://www.theguardian.com/cities/2018/oct/05/desire-paths-...
But that person could not possibly have predicted the future actions of everyone else, and therefore it's a mockery of moral reasoning to hold them responsible.
What are we holding them responsible for? Was something amoral? They clearly made the first decision for better or worse. Now if there was a sign that said "No walking on the grass", they clearly violated something.
And that's just it. Humanity has carved a path that says, "Make as much money as possible with the least amount of effort."
These machines have taken something we already know is bad, the 90 day, quarterly earnings cycle, which had already obliterated our long-term thinking down to nanoseconds and we want it even shorter.
We have machines moving virtual money around corporations that use money to move real material around the earth -- and beyond.
That is the path we are on right now in this very moment.
Who is going to put a sign up saying, "STOP! Humanity Restoration in Progress. Please choose another path."
Every organism since the first one has been focused on its individual survival. Interested only in the propagation of its own DNA.
And yeah, humans are still doing that. Everyone who took that path. Everyone who keeps taking that path of survival of the fittest.
We need to start thinking of the survival of the all of us.
https://en.wikipedia.org/wiki/Path_of_least_resistance
The path is optimized for A) Where people enter and B) Where they are going. If there is no purpose. A path is simply a path, and ascribing meaning to it is fruitless.
Imagine a groundskeeper.
Imagine a small box where every day at 6 am, an instruction on paper is written and the groundskeeper checks it.
Imagine a network of voters who vote on the instruction to be placed in the box. Their wide variety of votes will be automatically distilled into the winning instruction.
The groundskeeper dutifully clears a path, shouts at people to stay on the path, fills in errant paths.
Finally there is one path.
Who made the decision?
I only bring this up to suggest “emergent” decisions is a vapid framework to analyze this. Every phenomenon involving the aggregation of different agents is an emergent one, whether it’s an emergent system of politics leading to regulated actions or an emergent system of market participants leading to a price.
This doesn’t distinguish a situation of high regulation from a situation of low regulation.
A decision to follow the herd and a decision to take the easiest road are still decisions.
You might say that a person made a decision to break down blades of grass to create a path and that others chose to take a path already created, but they are still making choices that they are responsible for.
Answer: All of them.
- Whoever gave cachet to certain characteristics that are present at specific points of the grass
- Whoever made it desirable or necessary that passersby reach a certain location at the periphery of the field
- Whoever set up restrictions to entry at certain locations around the periphery of the field
- Whoever made any changes to the field (including non-human intervention)
More generally, culture at large and historical precedent.
—
Imagine a green field of grass.
There are minor differences across the field, but some of these differences can be noticed: a slightly darker patch of grass, a slightly yellowed patch of grass, some stones of varying sizes, a section that is slightly raised from the rest of the field.
Around the field, other things can catch the eye: there's a road to one side, some trees of varying sizes, one currently in bloom.
Over the seasons, the profile of this grass changes. At some point there was a park bench to one side of the field; later, it was moved to another side. There was once a storefront near to one side of the field.
Fairytales are useful, but they often hide the real truth.
Who is the man holding the smoking gun when algorithms collapse? The business owner is legally left with the responsibility, but finding who programmed the algo to make a mistake is useful & vital.
If the bridge collapses you get sued and your insurance pays out. You probably don't get more insurance.
This is not a perfect system, but we need several elements of it for algorithms. We need the ideas of tolerances, confidence intervals and analysis - certification of fitness for purpose must be limited and constrained to have value. We need professional certification that enables employers and the public to recognize experience and capability. People have confidence in pilots because of their uniforms, qualifications and the checks that they undertake (training, retraining, blood testing, medicals) on a regular basis. As we have seen with the 737 max, when these things breakdown there's trouble.
I don't think we can say this anymore. I wouldn't say that the makers of AlphaGo were the decision makers for any of AlphaGo's moves. You could perhaps say the players of the games AlphaGo trained on were. When it comes to AlphaGo Zero you can't even say that. At most you could say that the people were policy makers or tweakers.
It didn't spontaneously appear and start playing by itself.
So reference to AlphaGo doesn't make sense here.
Plus, deciding to delegate decisions (to algorithms or anything else) is not the same as deciding through your very own direct perspective.
It's true that automated trading is fast, and that means that things can go off the rails fast. And, it's true, that has led to some spectacular news-making events. But, for that very reason and others, most trading firms keep a very close eye on what their auto-traders are doing. Closer than they ever kept on their pit traders.
I briefly worked at a firm that did both HFT and pit trading, supporting one of the teams in charge of monitoring both groups. And what I saw was that the company was generally much more in control of the algorithmic trading. It was more heavily instrumented than a human could ever be, with humans (who were themselves licensed traders) watching the computers very closely at all times, as many layers of automated fail safes that would kick in if things went a little screwy.
By comparison, the pit trading was kind of a mess, and the team was frequently having to scramble to amend all sorts of human errors largely stemming from things like miscommunication and straight-up bad handwriting. Beyond that, I'm sure that, compared to the folks operating the auto-trading systems, the pit traders never had as firm of a real-time grasp of the finer subtleties of their positions, simply because nobody's able to do math in their heads like that.
Which, I think, is why spectacular events in the markets stemming from speculative trading going off the rails aren't really any more common nowadays than they were in the days before high frequency trading. Yeah, computers created some new ways to screw things up, but they also mitigated some existing ones.
The net effect is not nearly as bad as journalists make it out to be. But, of course, playing to people's fear of the unknown has always been the more lucrative route for journalists to take.
All of those people are gone. No need, we have automated much of this deal cycle. Your best example is to take a trip to your local exchange floor, which used to be littered with runners, assistants, floor traders, etc. It's eerily quiet these days, as most trades are executed by computers.
What I fear the most is a silly bug causing a crash in a way no manually triggered series of trades would.
Banks are run by smart people and they worry about the same things. There's dedicated systems checking to ensure algorithms aren't running away. All the big banks and brokers have been buying them for a decade or so. Nobody wants a crash and the algorithmic traders are mostly using a line of credit from the bank with which to do their trading and the bank usually take a commission on the profit so the banks have a strong interest in seeing them make money as opposed to crash the market and disappear into bankruptcy proceedings. Of course the government also swooped in and saved the day by mandating risk checking systems a couple years after everyone who cared was already using them (thereby solidifying the status quo for better or worse).
We've had flash crashes, which aren't in peoples interests, they still happened. I don't think its reasonable to hand wave it away, because "smart people". Programmers are supposedly smart people,I've yet to come across a perfect bug free program.
Human based checks are easier to subvert than automated ones.
>We've had flash crashes
Which is exactly the problem that most banks and markets try to prevent. Any system that uses what other people are doing as a strong enough input is going to be susceptible to this to some degree. If you try to buy too many shares of something or for too disparate of a price compared to the market the bank will almost certainly kill your order. The exchange might kill it too. Depending on your contract with the bank and exchange they may kick your session for the day if you do it too much (once may be too much). The limits most banks place on what you can do mean that runaway algorithms can really only be a slow tug in one direction or the other which is no big deal.
I disagree. They're easily subvertable, but in different ways.
"Which is exactly the problem that most banks and markets try to prevent"
But they haven't succeeded so far. Further, as the saying goes, history never repeats itself, but it rhymes. It's all well and good doing all these automated checks based on past issues, that's not going to help with the next issue that isn't enough like the one before.
A person might consider giving you a bit of wiggle room on the exact letter of the law, but can also ensure you comply with the spirit of the rules.
See also, Knight Capital: https://dealbook.nytimes.com/2012/08/02/knight-capital-says-... , where a bug cost it $440 million. It's already been seen on a small scale.
A collection of individuals and/or systems giving raise to a metasystem, a superorganism, does not and can not exercise deliberate control over it: the metasystem lies on a higher level. Not to mention as this thread - in its totality - surely proves, that individuals in many cases are completely unaware of the layer of control above them. What does an individual bee know about the colony?
What is "dangerous bullshit" is wishful thinking. Nation states have been with us for milennia. Corporations for centuries. Have we learned nothing that we still entertain notions of (human) global control?
The second part of that is that the human at the top can only make decisions on what is visible to them. With fairly opaque ML systems the only thing easily visible is the results. Seeing how it came to that result is much more challenging. So we will have humans at the top that can only makes decisions about the results and not the means. Which is the part many, including myself, find troubling.
A reasonable metaphor is someone firing a gun in to the air: they can't claim not to be responsible for where the bullet lands despite having no real control over where it's coming down. If you don't want that responsibility don't pull the trigger.
If an algorithm does it, there's no final line of defense. And software is easy to scapegoat because it's murky. You can claim "we didn't know the algorithm would do that," but it's harder to scapegoat your employees when there's emails on record showing you specifically told them to enact a social harm.
Who decided to make George W Bush president? I highly doubt you can assign responsibility to either a single individual or even fully enumerated clear subgroup of individuals.
Id also say that assigning moral responsibility to a group of people that large is really odd as well.
Edit: Just have a look at the disiplinary notices of some of the exchanges to get a feel for this. For example, here are notices for Nasdaq: https://www.nasdaqtrader.com/Trader.aspx?id=NDisciplinaryAct...
This is not true in case of machine learning. Yes, there was a person or people who designed the model, gathered training data, trained the model and deployed the result. But the decisions the model is making as part of its function are not made by these people.
Most of the time, people who trained the model cannot even tell you why the model is making the decisions it is making. Tractability of ML models is currently a very hot topic in the field and progress is being made, but a lot of currently deployed models are inherently intractable.
Seriously, the responsibility lies with whomever made—or okayed—-the system. One-off things, especially if they are hard to foresee, might get a partial pass. Otherwise you ought to be carefully testing and evaluating your own system instead of blaming gradient descent.
And generally speaking these are very different people, but that is besides the point. I am still unsure why we are talking about testing and evaluating, as if we are only talking about failure cases.
If I set the alarm clock for 7AM and it starts ringing at 4AM, the fault is with the clock designer. But if I set it for 4AM and it starts ringing at 4AM, it is somewhat silly of me to start blaming the clock designer for my disrupted sleep, isn't it?
And we are not talking alarm clocks, we are talking about systems that make complex decisions based on hugely multidimensional data. Decisions that we provably cannot explain. There is no amount of testing and evaluation that can ensure that these systems are error free. But when they work, they do work well and provide a competitive edge over human analysts.
> if everyone hates the current system, who perpetuates it?
And Ginsberg answers: “Moloch”. It’s powerful not because it’s correct – nobody literally thinks an ancient Carthaginian demon causes everything – but because thinking of the system as an agent throws into relief the degree to which the system isn’t an agent.