My Half Workday as a Turker
cs.cmu.edu
cs.cmu.edu
For example, say you are broke and wandering the streets of a big city and really want to buy a bus ticket to another city for $20 or whatever. No matter how much your net worth is, you can't get that $20 easily. You could try asking various businesses to do some work for cash, you could try a song and dance routine, etc, but your choices all amount to various forms of begging.
Meanwhile you can be sitting in an office somewhere and make that money in less than 3 hours even at minimum wage, even if you do little or no work. You can even sell something on craigslist if you’re home. I suppose in desperation you could sell blood plasma, but that’s one of very few lifelines.
Nobody thinks $20 is worth much, but when you don’t have it, it’s very expensive indeed.
To me, the root of the problem is whether you are resource rich or resource poor. So the web, by virtue of being intangible, is almost quintessentially resourceless. The trend seems to be lower and lower wages for increasingly onerous labor. In other words, if you have money, you get a real world return greater than the value of your money. But if you don’t have money, then getting it requires an expenditure of resources and effort larger than the value of the money itself.
At some point in the near future, acquiring money will be so expensive from a labor standpoint that it will be cheaper to simply do things yourself and live outside of mainstream society by bartering goods and services. This really bothers me, because that shouldn’t be the goal of progress. The paradox is that even though every new fiverr and mturk create more jobs, they lower the value of work. Right now this is affecting developing countries by creating a race-to-the-bottom economy, but the futurist in me looks all the way to the end and sees how so many jobs today (especially the non-production ones like administrative/clerical work) will eventually be automated by technology and thereby make the purchase of capital through labor even more expensive.
Does anyone see a way out of this? Did I miss something fundamental?
I'm not advocating a mass die-off of human beings, but any other means of limiting the supply of low-quality labor would have the same effect. If some sort of back-to-the-land subsistence thing became popular among the youth, those who didn't partake might find that a McJob paid more by virtue of there being fewer people willing to do it. Sort of a "shrug" by the lower-class Atlas instead of the elite.
The turk is so rarely updated there is probably some kind of disruptive startup opening for someone to set up a middleman service paying an hourly wage to carefully tracked humans.
This might put a stop to the horrible turk habit of claiming a 20 minute survey only takes 2 minutes or writing some kids term paper only takes 30 minutes, which is technically true if you spend no time thinking about it and can type at 250 WPM, but for almost everyone else its a bit unrealistic.
Some communities have sprung up on reddit to highlite great HITs but they're very realtime and seems like as soon as a great HIT is posted to the reddit group, in seconds its gone, used up. A better marketplace dynamic could be set up. Clearly HITs and turkers are not really commodites after all in practice, so commodity market tools do fail. Tools that better match the market would be better for all participants, and the market maker skimming off the top, so its mystifying why they put no effort into it.
This won't become popular because it means accepting a drastic decline in living standards: you lose Internet access, public transportation, access to a variety of goods, the rural area you move to might have a corrupt government or an underfunded police department -- why do you think gun ownership is so popular in small towns? -- you experience a lot of social isolation, etc.
This idea that people should move "back to the land" can only originate from the deepest misanthropy and ignorance of the plight of the lower classes -- or perhaps the most starry-eyed technocratic utopianism, if you think that society can somehow provide poor people in disparate areas with a modern lifestyle -- and I'm not holding my breath. Of course, we could just kick poor people to the curb like this; it seems to be our policy already, city councils won't approve new housing for them, they won't approve the construction of a Wal-Mart, expansion of transit services is always opposed by a chorus of veiled racists, etc.
That comment plus your handle ('presidentender) might get you a visit from the Secret Service ....
Also, what land? There isn't a vast reserve of untapped productive land lying around. It's all owned (capital) by someone who won't want you farming on it.
The only humane approach to too many people is a low birthrate movement, which requires universal free contraception. Something which a lot of Americans don't want to see happen.
That's not a solution, that's another problem. Japan, Europe, most of the USA, and even parts of Africa already have birth rates so low that they're below the replacement level. This causes workforce shortages, requires mass immigration, and causes several new problems in both host and guest countries. Japan itself is in a nose dive. Adult diapers outsell infant diapers. Less people = less customers = less money = less jobs.
"what land? There isn't a vast reserve of untapped productive land lying around."
That's what I thought. But there is. It's in Africa. A large swath of land the size of France got grabbed up during the 2007 Great African Land Grab which no one seemed to notice or care about other than the Oakland Institute. Communal lands were bribed and bought up for as little as 18 cents a hectare with the promise of infrastructure improvements and jobs. Few of which came into fruition.
No, it doesn't require that: without universal free contraception, economic development and, particularly, strong social safety nets reduced birthrates in much of the developed world to below replacement levels. Note that this is less true of the USA than much of the modern developed world because while the USA has good aggregate economic output, it has fairly weak and unreliable social support systems for a developed country.
People have more children when children are your insurance against poverty due to age, disability, etc. -- as they have been for most of human history. When that becomes less true, they have fewer children.
Essentially, when we get disconnected from 'the simple life', we glorify and idolize it.
Having come from back-woods, subsistence farming to a white-collar, higher education job, I can tell you that (a) very few people understand the amount of stress and work required to survive in subsistence farming, (b) seriously, stress; you can starve, or (c) how much it sucks to do without modern conveniences like computers, internet and electricity.
Ever wonder why current day subsistence farmers aren't well known for their poetry or art? Because they spend all of their time finding and growing food.
In the future, I think we'll see a combination of McJob and responsibility for one's own future, much as we've seen for every generation before. The more things change, the more they stay the same.
The way I see it, automated jobs will still add real value to the economy. Perhaps everyone can receive a share of that surplus through some form of universal income. At that point, assuming most of society's economic value creation is automated, monetary compensation could be completely decoupled from labor, yet everyone can be better off since everyone would be resource rich.
The foundation of the birth of modern socialism in the 1840's was based on the basic premise that technology would drastically increased efficiency and grow the economy to a point where redistribution could eradicate poverty and reduce the amount of labour necessary.
To your specific suggestion, variations over this was the viewpoint of some of the earliest socialist ideologists, who believed it possible to transform society through example and appeals to decency and charity.
Marx was one of the earliest to criticise that view strongly.
As early as the mid 1840's he made the argument that one of the main sources of eventual downfall for capitalism would not be some sudden enlightenment of the elites, but that capitalism would be too successful (the Communist Manifesto starts with a number of paragraphs gushing over the advances that capitalism and the bourgeoise have brought - Marx saw the development of capitalism as absolutely essential for progress) and eventually lead to over-production and under-employment at the same time, causing massive social upheaval and eventually leading to revolution if (though Marx also argument strongly that it was a when, not if) the ruling classes refuse to redistribute voluntarily.
The sad part is that his conclusion is only incorrect because the ruling class figured out (more plausibly, stumbled into) a strategy for doing precisely the minimum amount of redistribution required to avoid revolution.
http://www.guernicamag.com/daily/anya-groner-the-heart-you-s...
Another part of this is globalization -- you could argue that most people in the US (for example) are quite well off (as contrasted with complete destitution, starvation and lack of clean water) -- and one way to stave off revolution is to simply live on the other side of an ocean, and fill that ocean with friendly nuclear submarines and hangar ships.
But there is no guarantee such a scheme will last forever either -- as you'll likely still have people working in factories, and those factories will be producing (among other things) the weapons used for maintaining the status quo. At the very least it's hard to see how any industrial production could be made immune to strikes and sabotage, if the workers were desperate enough -- and that would likely result in a transfer of wealth (by no means is a "fair" outcome guaranteed, but change seems very likely).
I think it is too early to conclude that the revolution will never arrive. Be that just a small one (redistribution of wealth in the US for example) or a larger one (redistribution including the entire world).
"The problem is that the worker who is going to be out of a job doesn't own the company that buys the robots to replace him/her. Ever heard of a lights out factory that has all the former assembly line workers at home with full salary? Didn't think so."
The way capitalist societies are structured doesn't really allow for income distribution that way, the way I understand it.
In the fully-automated future, these societies will violently implode if they aren't reconfigured to provide people with something to do and something to eat.
That's pretty much true-by-definition when it comes to "capitalist societies", but most advanced modern societies aren't really capitalist, and while the course hasn't been strictly monotonic, they've been overall getting farther from capitalism since the late 19th Century heyday of capitalism.
Port containerization went a bit like that. Of course, that was back when America had labour unions.
This is natural evolution, IT wages bubble will burst because anyone with the proper education in any place in the world with lower expenses will be able to do it for less.
I hope the same happens with education, internet will change humanity in ways we can't imagine right now.
1-Globalization: technology and governments based in short term economics allow work to be made where is cheaper -> wages will be similar everywhere in the world in the near future.
2-Supply and demand: technology decreases the demand of human labor (automated work) and increases the supply (mturk, fiverr, freelancer, etc) -> wages decrease.
This is Paul Krugman's take on it:
http://www.nytimes.com/2013/06/14/opinion/krugman-sympathy-f...
How does that solve anything? Sounds like a pretty bad side-effect or is that really an objective with basic income?
Increasing the cost of everything is an unfortunate side-effect that will hopefully not be very big.
A Basic Income would help solve the problem of poverty and climate change, and should be a human right.
To me, it doesn’t make sense to say that money is non-linear or non-constant. Money is not a function of time, therefore it does not make sense to describe it with time-dependent terms such as “linear” or "constant”. Inflation, the change in the purchasing power of money over time, yes. Money, no.
Money is simply a token that we use to represent the intangible concept of value. I think your comment seeks to address a change in what individuals, and by extension and on aggregate, societies, fundamentally value over time.
I’ll return to that in a bit. First let me address the problem with your example: It ignores the compounding returns to performing similar actions over time. Economically: Human capital accumulation. Colloquially: Experience.
When you get your first entry-level salaried job, you are actually a liability to the company. You come in with a low level of human capital - the company is basically paying to train you, all the while putting up with any potential mistakes due to your lack of experience. Why would a company do this?
By hiring you, the company is making an investment in your stock of human capital. It takes on some risk up front in return for some amount of reward in the future when you are wiser. As you gain experience over time, (ideally) you perform increasingly valuable tasks with a higher efficiency. You have accumulated the one thing youth (or old age for that matter) cannot buy for itself: time.
Getting paid for 3 hours for "little to no work" is a drop in the bucket compared to 10 years of solid, salaried employment.
By comparison, one-off jobs which might net you $20 in a couple of hours are a completely different situation where a low level of human capital is not purchased by means of a salaried contract. Rather, it is rented with a much shorter time horizon in mind, generally for menial tasks that require little to no experience.
To connect this idea back to the larger debate of “lowering the value of work”: I think you need to specify what kind of work you are talking about. I agree that the value of human menial labor is constantly decreasing. Why?
The supply of physical, menial tasks (washing dishes or mopping floors) is determined by the inflows of the human birthrate (more humans=more eating=more dishes to wash) and the outflows of automation (dishwashing machines), among other factors. Automation adds value by adding hours to our lives that would otherwise be spent washing dishes. For menial tasks it is generally more cost efficient. Thus the outflow outweighs the inflow (decreasing birthrate) leading to a lower supply of menial labor.
I don’t think we will revert to a hunter-gatherer bartering society in the future. Markets and specialization are much more efficient forms of raising standards of living, which I define as the ultimate goal of “human progress”. I think we will eventually need to reexamine our definition of “standards of living” as we move away from the realm of the physical to that of the virtual. Perhaps one (sad) day food will be unnecessary, as we redesign our bodies to subsist on a different form of energy, or do away with physical bodies altogether. This standard of living does not take into account food, shelter, transportation, etc. – the fundamentals that we strive to improve today through progress.
Anyways, enough with the science fiction and philosophizing. The fundamental lesson here is that the market gets what the market wants. There are a lot of human beings, so menial labor, both physical and mental, is in high supply driving down its relative value/price. Technology is eating jobs from the bottom up. Increase your human capital so that your brain is more valuable than a charged hunk of steel. Be the person who knows how to operate the machine and you will never have to beg for a $20 bus ticket.
I've had a theory that out in the world somewhere is someone selling a guide to using MTurk to create article summaries and the guide suggests the price range of $.25-.50 because for the entire time that I've been on MTurk (many years now), every time I come back the price range is the same with small fluctuation.
Past the relatively low amount of summaries that are posted daily, money can be made by watching the right forums where people post high paying HITs (the name of the most popular forum eludes me at the moment) that are typically 5-15 minute surveys that pay .5-3.00 each.
By far, the biggest barrier to making a decent wage for me was the number of available HITs. The ones that pay over pennies each are few while the ones that are horrible time invests are everywhere. If I had an unlimited number of summary HITs, I could have made well over minimum wage.
[0] Made about $450 over the time, but this was sporadic, random working periods.
It sounds like good practice in concise communication.
[edit: I now realize that that's not enough data, we'd also need how long it takes to write the average hn comment, and how long op took writing his :-) ]
In academic research it is especially important, and made more difficult by the complexity of the subject matter. The most successful researchers can succinctly explain their work (problem, significance, methodology and findings) to a layman. That's how you get funding. This also applies to startup founders seeking investors.
It might sound strange, but I find certain kinds of survey type simple tasks incredibly relaxing and some of the HITs clearly connect to what seems to be interesting research.
Pretty much everything the author says here is spot-on and has been for a long time.
MTurk is positively flooded with HITs for generating fake reviews for products and fake content for websites. The best paying HITs by far are translation tasks - transcribing Arabic and Farsi seems to pay the best ($0.30 or more per minute of transcription).
I aim for academic projects with preference for things which require qualification (simple math / reading comprehension tests usually) as they generally pay quite a bit more.
I've earned an average of $0.27 per HIT and have never had one rejected.
Some of the more interesting / unusual tasks I've seen:
* Pinpoint various joints (shoulder, elbow, knee) in each frame of a clip showing a baseball player swinging a bat.
* Drop a pin on an estimate map location based on scenery shown in a short video clip (This was long before MapCrunch/GeoGuessr).
* Manipulate the camera in a scene of flat-shaded objects to bring them into 'correct' perspective.
* Choose 'preferred' structure designs (little houses) which appear to have been created by some sort of genetic algorithm.
I get distracted by categorization jobs, like look at this blueprint and if there is a blueprint number in the title box, enter it. Well, I end up admiring the print. Oh look a giant gear for some kind of mining thing. Why, what an interesting looking architectural drawing.
Similar thing happens with categorize document numbers. Apparently some municipality decided to turn millions of paper scanned pages of contract bids into piles sorted by contract, so I'd end up sitting there reading all about bids to put in some drainage culvert. Who would ever have guessed theres so many steps and processes and backfill and compaction are critical and theres so many inspectors... I mean you can half A a drainage culvert in like one paragraph but if you want it done right its 50 pages of PDF.
These jobs are probably impossible without an excellent short term memory or dual (or more) monitors.
The ones that annoyed me the most were "leave blank if no number" so I'd scour the document for what seemed like forever just to make sure. Inevitably leading me to becoming fascinated by drainage culvert design.
Its very D+D like. Whats behind this dungeon door? Oh, I see, a blueprint of a giant gear from some mining equipment. Fascinating. Like archeology but I'm getting paid for it. Well, not much pay.
I found this quite amusing. You know why this HIT was approved? Because requesters are also punished for rejecting too many HITs! He also would have been banned had Amazon discovered the scheme he was pulling.
That suggested that there may be an interesting 'toolware' market for Turkers. For example, document editing; Lets say we build a simple document editing platform (copy editing not typesetting) which includes components that are dropbox like (shared storage), Google docs like etc. And design a workflow around that toolset. So pick a HIT, get a document id, it shows up in your shared storage, use the included tool to edit it, when you're done, click 'done', and have the whole thing resubmit back for evaluation. That might allow you to focus on editing and not get hung up on a bunch of getting it done issues.
Not sure how practical that is, but if there are common Turk workflows it might be useful to build some common tooling for them.
In the early days of Mechanical Turk (2005-06), when almost all the HITs were seeded by Amazon themselves and were of a few common types that kept being replenished, there were browser extensions to greatly improve the UI for the common tasks (the default UI is very bad). One of the more profitable tasks was fixing the location of addresses on the new-defunct A9 BlockView (a product similar to what Google would later introduce as Street View). They'd present you about 10 photos of a streetfront that were supposed to be near an address, and you were supposed to pick the one centered around the address (e.g. the store or house entrance). Or else indicate none of the above, if the alignment was seriously off so that the correct address was out of the frame. If you used the regular website UI, you could make maybe $3-4/hr, but most of this was tedious clicking: accept HIT, click a tiny radio button, scroll down, submit HIT, request next HIT, repeat. The browser extension implemented the obvious UI improvement: use the images instead of tiny radio buttons as click targets, and then auto-submit and request the next one on click. I think in one version you could also just hit a number 0-9 on the numpad instead of clicking. With that extension I was making $20-30/hr for a little bit.
Amazon Mechanical Turk: Gold Mine or Coal Mine?
http://www.mitpressjournals.org/doi/abs/10.1162/COLI_a_00057
Rather than crowdsourcing being a way to find cheap writers, content business should see crowdsourcing as a way to find a diverse pool of writers who have different interests or knowledge, who can dip in and out of work, and then pay them fair money for their time.
I pay $1 per post for writers to write 50-100 word posts that follow a very specific format. We provide feedback to the writer to try to make them make money faster. The way I figure it is, if the writer makes money then I can make money. If they stick with you, you don't have to train new people. People won't leave after half a day.
We have already paid out thousands of dollars and our site is growing fast. Writer satisfaction is high. We currently have a waiting list of hundreds of writers wanting to sign up.
I'd assume anyone that can write well, should be able to make at the very minimum 30/hour writing -- That leaves on average 2 minutes/newsline. I suppose the lesson is that there are many skilled poor people out there.
(Note, this isn't meant as negative criticism, just some observations)
http://www.forbes.com/sites/susanadams/2012/04/26/is-journal...
Journalists earn (for ~2000 hours/year) between 15 and 28 (without a graduate degree), while lower level CS started at around 25/hour. So I was clearly wrong, but it would also appear journalism isn't especially low paying either?
http://www.poynter.org/latest-news/mediawire/202644/gov-stat...
Seem to mostly agree.
Read more about the tool: http://www.cis.upenn.edu/~ccb/publications/crowd-workers-pos...
Try out our Chrome plugin and our web site: http://crowd-workers.com/landing
The most frustrating jobs i've ever performed on the platform were usually approx 5-10 minute tasks that seemed to be decently paying and straightforward, asking you to categorize items based on some arbitrary but seemingly simple system, but later all 5 or more submissions you might have made could be blanket denied / payment refused. In that case not only do you not get payed, but having HIT submissions rejected really seriously hurts your reputation and what jobs you're allowed to apply for on MT; often many more high paying jobs on the system use high acceptance rate and flawless work history as the lowest bar for entry, and so MT workers can be screwed over incredibly by having work rejected through no fault of their own.
To not get seriously screwed newcomers are somehow supposed to know which types of HIT's to definitely avoid that could kill their rating, but without enough information to actually make that judgement. A task might seem easy and give the illusion that it's okay to send 5 or 10 submissions in a row, only to later have them all rejected and serious reputation damage done.
I think it's interesting but also really sad how this mirrors a lot of the unfairness and power imbalance in the larger working world as well. Reputation is everything, but if you just happen to get unlucky to start out your career working for clueless bosses / clients with unrealistic expectations, there's a good chance future job and career prospects are going to be seriously hindered, if not completely derailed.
Transcription work was usually very well paying i found, but doing one long transcription task for a $30 payout is a huge risk the way things were set up. Never mind that you worked for 4 or 5 hours on it; whoever assigned the job can reject it for whatever random reason they want and you get nothing. As nice as it can be to earn a nice wage for more high-skilled tasks, the possibility of being denied any payment for a days work with little recourse is really frustrating and demoralizing; the way Mechanical Turk is structured if anyone's taking a negative financial hit for work done, it's almost without fail going to be the worker.
Though this CMU professor obviously put some work into this write-up, i don't see how somebody can possibly get an accurate picture of what it's like trying to earn an income on one of these platforms by just devoting 4 hours and extrapolating based on that really limited experience. It is really easy to miss all but the most obvious and glaring problems that way, just from not sticking around long enough to even run across them.
There are plenty of MT workers who have spent much more substantial amounts of time on the platform, that are much better able to communicate benefits and pitfalls for the everyday worker. They might not be professors at prestigious universities like CMU, but having a decent amount of hands-on experience should really be the low bar for discussing pros/cons of the platform seriously; just as one example assumptions about longer-term take-home pay in the write-up were pretty naive: many better paying tasks (especially web browsing tasks and surveys paying more than a pittance) are often in limited supply and cannot be repeated either at all, or only a few times per day / week, etc. Even the best employers providing the most decent fairly-compensated work come and go, making a steady healthy income sometimes very difficult to achieve even in the best of circumstances.
Mechanical Turk has experienced a lot of success from the start largely because they created a service that makes it so easy for virtually anybody with internet access and a basic computer, to work from home and perform simple tasks / other types of work when and wherever they want, and actually get paid for it.
By any decent standards of developed-world countries the pay is shit though, and worker protections and benefits are nonexistent; workers can be denied payment and even have their reputation ruined for no good reason, with little in the way of an appeals process. But even now services like MT are still a pretty new thing. Maybe in 10 or even 20 more years most of the bugs will get worked out, and workers of every skill level can be a on more equal footing when negotiating with employers, settling disputes etc.
[1]turkopticon.ucsd.edu
Seemingly arbitrary rejections always seemed to be a much bigger problem by my estimation, especially for workers already somewhat invested in the system, and needing to maintain approval ratings to even qualify for higher-paying types of work.
If Amazon hasn't implemented anything like a way of expunging old rejection records, it would be a huge benefit to their workers to have some sort of reasonable, semi-automated way of doing so, at least after enough time has passed, or enough more recent positive approvals have occurred.
I'm really out of the loop though, and don't know much about the current state of things. It would just be really nice to have some more in-depth write-ups on the MT worker experience. The article was great but from first-hand experience i know there is a lot more ground to cover.
Senior project / undergrad thesis anyone? Researching / investigating MT from a user interface, user interaction design side of things, giving recommendations for improvements to the worker experience, for HIT providers, etc. would be very interesting and potentially very beneficial for the platform.
I bet you'd learn some great stuff and see some crazy software
Looking at your posting history, I can see that you often take the time to write thoughtful comments. Please keep it up.
I hope you're trying to be on the workers side but either way this is a terrible thing to say. The way you look down upon people trying their very best to make a living is not helping them at all.
It's also interesting that the linked page has two scroll bars (tried both Chrome & FF).
If nothing else, Turkers are good at making sure your data collection instruments are working the way that they are supposed to.
Do you ask any 'trick' questions that are the cognitive equivalent of a captcha?
I've never run into a serious college professor category of trick question.
Turk is a grind game if you don't need the money. If you're susceptible its going to be a little moth to flame for you and all that matters is your meaningless number goes up, or if you're easily bored like me, you'll turk for like 20 hours one week and then not think about it again, until this HN question, in fact.
I will say that any researcher type would be horribly disappointed by turking because I did it while watching TV, or drinking a beer, and I have a low tolerance not being much of a drinker. Speaking of beer, I am told there is, or was, something of a fad of trying to turk fast enough to pay for your booze. There is no way humanly possible to turk fast enough to drink at a bar, but cheap freshman class beer in a dorm is totally realistic. Can you pay for that six pack on mturk before you finish drinking it? So... I don't think anyone could get homework assignments done on mturk. At least if you want to pass the class or have graduated from grade school. The linked article example is far beyond anything I ever wrote for mturk.
The most insulting and infuriating turk jobs I ran into were not on the author's list, basically being paid to be spammed to. Here's 20 full page variations of a home equity mortgage spam, now pick your favorite one for a nickle. Or call this number and talk to this scammer for a minimum of 5 minutes and then come here and fill out a survey for like 10 cents, at least on the surface its for "quality control" but you know its really to give you the hard sell.
Life is cheap on the turk, and its full of work hits that pay the equivalent of a buck an hour or less, which is eventually super demotivating / depressing. You hear about cool high paying hits but end up filling out 20 page surveys about B.S. for like ten cents, and getting sick of it.
http://www.europeansocialsurvey.org/methodology/improving_qu...
The valuations were fair but the workers of Mechanical Turk may have a different view of how much something "home made" or "retro" may be worth (typically far lower than the object would be worth for someone in the market for said object).
The hardest thing is always task design, as the author discusses. It's quite challenging to create a task that will work in all browsers, across cultural and language boundaries, and to pay enough to incentivize the worker (while simultaneously little enough to make all the effort worthwhile).
The final effect is a bit dehumanizing on both ends. The workers feel alienated and mistrusted by the people requesting the tasks. The people requesting the tasks feel that the workers are lazy / "scammers" trying to cheat them.
We don't fully even know how to think about this kind of work yet - ideas like "hourly wage" don't seem to fit for someone working completely independently, at their own pace, possibly distracted or impaired.
Still, despite the trouble, it seems like one of the most promising fields for interaction design. There's the possibility to really improve peoples' lives (or make them worse).