Chelsea Manning: The Dystopia We Signed Up For
nytimes.com
nytimes.com
She thinks of that as dystopian because of the foreignness and inconvenience.
We, as technologists, can improve upon this by talking empathetically more with our users about how they interact with our technology, to help steer the product towards incorporating more human-focused designs. This could help avoid this Stranger in a Strange Land phenomenon becoming more prevalent, and avoid the negative psychological behaviors which disconnected individuals typically exhibit.
Please explain what you mean by "disconnected individuals" and what are the behaviours. Genuinely interested.
I've been hearing this a lot lately but I don't really know what it means.
I think most (all?) developers/designers try to imagine the experience of an average or "canonical" user while developing features. This is a natural way to try creating a product with decent usability. You can take this a step further with UX studies, focus groups, etc.
However, since "UX design" has existed in one form or another since before PC's were a thing, I take it that's not what's meant by "having more empathy for our users".
I suspect this means somehow tailoring/adapting our products to the wide variety of cultural, geographic, socio-economic backgrounds of users, or even their states-of-mind/existential experience. In that case, I think a UX designer would face a combinatorial explosion of constraints and/or features requests, which is not practical.
However, we do have a free'ish market, which seems well positioned to address this wide variety of needs. Moreover, the long tail of users is getting fatter as our population scales, so it is increasingly rewarding to target niche groups.
We can do some of it but most of the 'do no evil' goes away in favor of what the money wants to do. Even well intentioned designers, developers/engineers, entrepreneurs, etc get pushed/pulled in directions of the money and usually understandably. The money machine always gets what it wants.
A big problem with our society today is, the people with all the power (lever on the money) are the same ones selling us out to dystopian lack of control of privacy, rights etc. We are a system setup where the only good that can really come is if wealth, or someone wealthy, wants to see it happen. We are basing all our opportunities, hopes, rights on very few people that are actively getting paid to cut them down.
Inequality does us in more than we realize because it is easy to throw the people under the bus for the money, once you are in the money you aren't in the people by design. We live in a 'free' country, but combine that with the feudal/sharecropper setup at our dictator-like controlled companies we work for where corporate rights overpower individual rights on the regular, and well you end up with a recipe for dystopia.
The grandparent comment expresses a beautiful sentiment -- and one that I and many other colleagues share -- but I've almost never seen it implemented in practice; and on the 2-3 occasions over a 15 years of career when I seen it, it was on a smaller scale and with business owners who had a close relationship with all technologists in the "company" (I use quotes here because an organization with 10 programmers, 1 sysadmin, 1 HR, 3 sales people and 1 CEO can hardly be called a "company").
Truth is, many businessmen don't think of people like people; they remind me a lot of the young military boys controlling drones that do the remote killing -- they think of people like "assets", "targets" and such. This de-humanization helps them make decisions that many of us would find extremely hard to justify in front of our moral values system.
This limbo lets you make a mint, but it will end.
It's not like she was permanantly shut out of society. She was temporarily inconvenienced.
When she got out, she tries to fix up her accounts.
If someone walks up to a bank years after no activity and wants a name change (and maybe a gender change) on an account should the bank just do it? Probably not without 3rd party verification that the customer has to initiate by filling out some forms and providing evidence.
I'm just speculating but it could have been reasonable for the bank to have declined the request up front.
I bet if it was a woman that got married and changed her last name, it would be super simple.
Also, I don't think that's the full extent of her point here. She's just offering an example of how our pervasive reliance on computers (which often have poorly designed software) has caused trouble for her personally.
A system where only a handful of people have to wade through a bureaucratic mess with alarming regularity can get very dystopian very fast for those few.
Personally, I would consider having to forego food and/or shelter for an unknown length of time -- while a bureaucratic morass is being sorted out, because "the computer said no" -- to be much, much more than just a "temporary inconvenience".
The worst case is that you get handled by people because the computer didn't work, which is the same as if there were no computers.
We might not be optimized for human intervention into these processes, but that's because at the scale we operate at (hundreds of millions of users) it is physically impossible to provide human to human service for everyone.
You are for some reason assuming the article described a failed computer-aided transaction gracefully degrading to a successful human-aided transaction. But that is clearly not what it describes:
> Even then, human clerks and bank tellers would sometimes see the discrepancy, shrug and say “the computer says no” while denying me access to my accounts.
In other words, this was an unsuccessful computer-aided transaction that a service representative did not have permission to override.
If there had been no computers, the service representative would have successfully completed the transaction since there would have been no discrepancy.
So in this example, computer discrepancy + powerless human representative is worse than a representative in a world with no computers.
Edit: wording
If you replace a computer with a piece of paper with a different name on the account, it doesn't make it any easier for a service representative to override.
This isn't a computer problem. It has always been inconvenient to change your name.
A "service representative" (i.e., a real human directly in front of you) could have taken a look at that legal paperwork -- signed by a judge, mind you, and likely imprinted with an official stamp from the clerk -- that Manning was carrying and came to the correct conclusion rather quickly.
At the worse, the rep. might have had to get up, walk 20 feet, and have his/her supervisor also review and approve it. At that point, it would have just taken a few moments to update the information in the computer and the problem would have been resolved permanently.
At least in the cases the author describes, I'm not sure why this would be true. Especially once you consider that, over time, less skilled tellers get hired to fill the position of essentially trusting the computer to do most of the heavy lifting.
Why, for example, would a manager keep granting the same power to the teller to use their own discretion, especially in cases where their discretion contradicts the recommendations of the computer system? The potential to gracefully handle edge cases comes at the much greater risk of the unskilled teller screwing up and costing the bank money.
It certainly does: * Before computers, most of the roles of the computer were not fulfilled by pieces of paper. It was the teller-of-the-past who was the human front-end and identity verifier/calculator/UI/running policykit instance/etc.
* teller-of-the-past was necessarily trusted to use their discretion to a greater degree than teller-of-the-present
* teller-of-the-past could cache the results of an identity verification on Tuesday to use as the identity verification on Wednesday
* supervisor could bless teller-of-the-past to teach teller-of-the-past2 how to do an end-run around policy foo for edge case bar, where edge case bar involves a specific human or category of humans. (As an aside, the supervisor could do this more quickly than a team of third-party software developers can code, test, and deploy an interface for the same human or small category of humans)
* piece of paper could not lock teller-of-the-past out of completing a transaction with the customer
* even if teller-of-the-past decided not to fulfill the transaction, the equivalent of the computer's identity verification algorithm was probably someone else standing in the same physical bank location who could be asked to help complete the task
> This isn't a computer problem. It has always been inconvenient to change your name.
I think what the author is trying to describe is a problem of lack of human discretion, which is something that wouldn't exist at the scale that it currently does without over-reliance on computers. Gmail and plenty of other services don't degrade to humans at all, so I don't think this is subtle stuff.
Tell that to my friend who is presently without health insurance for his whole family because there isn't a human to help him sign up. His employers put too much faith in the computers and that faith is only going to increase until a critical threshold of people (and the right kind of people, that is not peons like me and him) are inconvenienced.
He's also sans a paycheck and delayed on configuring his retirement contributions, costing him thousands in the long run.
IMO the only thing that matters at the end is whether people _care_ that something like this is wrong and want to fix it, or if they treat it as as "outliers gonna outlie" so they choose not to care.
Without that level of concern, people who don't easily fit in to the mainstream find their lives increasingly considered less and less important.
If some random person walks into my bank and says they're me, but they've changed their name (and had surgery to change their appearance) I would want that person to be inconvenienced and go through hoops to prove it. I wouldn't want it to be easy for them to assume my identity.
having some trouble handling edge cases is not dystopian
In most literature it's a society that has a myopic view of people (ie, not dealing with edge cases) resulting in an oppressive society. One where fairness and free agency have been greatly curtailed by a stubborn insistence on a very simplistic view of the world.
1) The "edge cases" typically change slowly over time. If you are an edge case, it's not your turn this year and someone else's turn next year. Your circumstances define you, and are potentially reinforced by your "edge case" status. It's your turn every year, and your entire life becomes a living hell.
2) It takes only a small proportion of edge cases to get a lot of disenfranchised people. For example, 1% of people being edge cases would draw in over 3 million people in the US alone.
I have hemophilia, a bleeding disorder. Normally, its a nuisance that just requires I take medication on semi-daily basis. If I don't take the medication, I'll usually be fine, except if I have an active bleed, which will just continue to bleed, causing damage to my body, and can be life threatening depending on the location of the bleed.
I was recently traveling back home and I ran out of my medication. It's a specialty medication that you can't find in pharmacies. It normally has to be shipped to me. I had ordered some but it wouldn't arrive for another day.
I was having an active bleed in my torso, of all places, which is an odd place and distressing because its close to so many important body systems (the spine, major arteries). The thing about my condition is that normally bleeds are innocuous, but you never know. I always be on the safe side, because when I haven't its often meant months-long recovery.
Anyway, I figured no problem, I'll just go to the hospital ER and get some medication, which is what I'm supposed to do and I've done before -- but not in a long time. Most all major metro hospitals carry my medication.
I went to the hospital and told them about my condition and what I needed. They wanted me to take a CT scan, which I declined (I already knew what I was dealing with and just needed my medication). Then, they wanted to test me for hemophilia (I've had the condition since birth), a very expensive and lengthy test that takes hours.
After finally speaking with a doctor, he declined to give me my medication until I both consented to the CT scan and the test for hemophilia (despite me having official medical identification that explains my condition). I didn't want the CT scan because its ungodly expensive and exposes your body to a lot of radiation (100x an x-ray). You could clearly tell it was a bleed due to swelling, bruising, heat, and my prior medical history. And I've lived with the condition for awhile so I am very attuned to these problems.
I experienced what I can only describe as a similar type of hostility as someone coming in seeking drugs to get high (my medication neither makes you high nor is dangerous, except when I don't get it).
Curious as to why I was being treated this way, I pressed further. The doctor came back with a print-out from some medical "AI" software. It had not only determined what the proper course of action was to be (the tests before my medication), I was also informed that if the staff deviated from the "process" at all, they would risk being fired.
There you have it. All decision-making ability had been revoked from even the doctors. They were prepared to let me bleed for hours, while waiting on a test to confirm a condition I've had since birth rather than risk deviating from some computer-generated "AI" policy. Even more sinister, since it was a screenshot (done with print screen) of the windows-based software, I spotted something else: something called a "risk score". It was a gauge-like visualization that had rated me in the "riskier" category. No doubt this was also the case with patients who are drug seeking, and I hypothesize this was the reason for the apathy and apprehension I detected.
It kind of reminds me of the Critical Care episode of Star Trek: Voyager, where patients have been reduced to a "TC" -- treatment coefficient that determines who gets good care and who doesn't and is based on the person's accomplishments and value to society. Honestly, we're not far off from that.
What would happen if I came into the ER with a serious problem that required immediate treatment? Would they defer to this software and watch me die while waiting on tests? I know my condition is rare, I'm used to educating doctors about it, but the fact that the doctors seemed unwilling to even compromise and do the right thing was alarming to me, and also sad. Its like they have become customer service representatives of a large corporation and they know it, and so they have divorced themselves of any care or humanity at all.
I decided to leave and take my chances on waiting the day, rather than sit there for hours. At that point, I was in extreme pain and I honestly didn't know if the bleed would resolve itself or not. Worse case, I could always come back to the hospital. I guess.
The bleed ended up resolving itself, and I eventually got my medication and was able to treat myself. Ironically, a nurse called later the next day to "check up on me". My tests had finally come back over a day later and confirmed my condition, and also that my factor level was low. After I told her the story I've just shared with you, all she could offer was to "make a note in your chart". That no-one will be looking at, and likely not influence the medical software at all. I looked on the hospital web site for contact information to try and lodge a complaint -- there was none. I also tried calling the hospital's 800 number, which was entirely automated and offered no ability to talk to anyone. In case anyone is wondering, the name of the hospital is Riverside in Columbus, Ohio (part of Ohio Health). It's a huge hospital.
Anyway, this is my story about AI. Something to think about because as engineers we are the ones making these kinds of software and have an ethical and moral responsibility. We need to be aware of how people are going to use technology and actively work to prevent abuses, and hopefully build this into the software itself.
What kind of science fiction novel are we living in?
I wonder why (a) they couldn't just call your regular MD to verify your info, or (b) your MD could not phone in a prescription to the hospital's pharmacy?
The problem is my medication needs to be reconstituted (fancy word for "mixed") and is given via an IV (a shot).
I can and do both of these things, but with these kinds of medications they simply can't just give them to you -- they have to be "administered". That's a topic for another post, but it explains why they just can't give me the box of medication and supplies. Probably some kind of liability issue.
The miniseries on the Unabomber that just ended this week could not be more timely, even after 20 years.
Is it really that difficult to understand the subtext here?
Beyond that, I was answering a question about subtext, not discussing the article directly.
Your argument would still "apply" if Chelsea had cheated on a lover; "hey, she betrayed someone's trust so how can she talk about trust?" Basically, nothing's she's done invalidates or even weakens her points.
"Don't let anyone get your private data or it'll be leaked or mishandled -- trust me I know all about mishandling privileged data."
Which is "mishandled."
In this dilemma which trust has the higher priority? In your opinion it should be the US military, in Manning's case, it was not.
Her original intention, which has gotten somewhat lost in all this controversy, was to protect atheist groups in Middle Eastern countries who were being labeled as "anti-government radicals" and persecuted, in concert with US intelligence forces. I certainly agree with her frustrations there. But not her tactics.
She broke her employment contract and the rules of her security clearance, then bragged about it to the wrong person and paid an appropriate price.
I personally don't believe that saying one side followed all the rules when they have the power to make the rules has any weight as a moral argument
Chelsea Manning made us aware of it by leaking documents I deem to be in the public interest.
Between the US military and Chelsea Manning, guess who I trust more?
It's similar to a reformed/retired safe cracker is telling us which safes are hard to crack. The safe cracker would be in a position to know about how difficult it would be to crack safes.
The major difference being that the safe cracker possibly stands to benefit from lying to us in an effort to make their job easier (convince us to get safes that are in reality easy to crack). In the situation with Chelsea Manning, I'm not sure where you think that she stands to benefit from us following any advice that she's giving.
That safe cracker from example above would be ashamed, he couldn't even dream of something like that.
So basically, she saw the government's breach of trust and called it out, and is doing so again now.
It seems like either you don't understand what Chelsea did in the first place, or your argument is simply with Chelsea herself for any number of possible reasons.
Yes, this is where we are now. Fucking good.
It wasn't a well curated dump, much of the data wasn't whistleblowing.
If you compare how Chelsea handled it vs Snowden they are worlds apart. Chelsea wanted to be a whistleblower but IMO behaved like a leaker/dumper.
Of course intent does matter but, even though I'm very pro-Chelsea, lets not pretend Chelsea behaved admirably in how she handled the leaks.
Heck, government is even trying to stake claim to outer space:
Even worse, imagine if we stumbled upon something like a galactic council of sorts. They'd nuke us to oblivion for being the savages we are. (Actually, if they are more civilized and technologically advanced than us, it's more likely for them to only forbid us from expanding any further, but you get the general idea.)
With a criminal record and fresh out of prison, she's not going to have many (decent) opportunities available to her so perhaps this is one way to make the most of her situation and/or experiences. I'm sure there are plenty of organizations that would pay for this.
She was named as a visiting fellow at the Harvard Institute of Politics, literally today.
Maybe not before becoming a public figure, but she's done a lot of public writing since, including the last couple of years as an official contributing writer for Guardian US, having written a couple op-eds for them before that.
Prison does often give one time for studying, reading, and writing if one chooses to use it for that purpose.
I was frankly floored by Manning's calm thoughtfulness.
[1] https://theintercept.com/2017/05/10/intercepted-podcast-jame...
[1]http://content.time.com/time/business/article/0,8599,1954643...