Anyone else witnessing a panic inside NLP orgs of big tech companies?

I see where you’re coming from, but is this really the main source of the inequality?

Based on numbers relating to workers’ diminishing share of profits, it seems to be that the capital class has been able to take a bigger piece of the profit pie without sharing. In the past, companies have shared profits more widely due to benevolence (it happens), government edict (e.g., ww2 era), or social/political pressure (e.g., post-war boom).

Fwiw, I think that the mid-20th century build up of the middle class was an anomaly (sadly), and perhaps we are just reverting to the norm in terms of capital class and worker class extremes.

I see tons of super skilled folks still getting financially fucked by the capital class simply because there is no real option other than to try to attempt to become part of the capital class.

And it doesn't mean that the replacements will be much better, or even as good as the Humana they replace. They will probably suck in ways that will become familiar and predictable, and at the same time irritating and inescapable. Think of the outsourced, automated voice systems at your doctor's office, self-checkout at the grocery store, those touchscreen kiosks at McDonalds, etc.

I already find myself wanting to scream

> GIVE ME A FUCKING HUMAN BEING

every now and then. That's only going to get worse.

Before mechanical alarm clocks, there were people paid to tap on windows to wake them up.

As for human translators, the need for them far, far exceeds the number of them. Have you ever needed translation help? I sure have, but no human translator was available or was too expensive.

I have been doing NLP since 1993. Before ca. 1996, there were mostly rule-based systems that were just toys. They lacked robustness. Then statistical systems came up and things like spell-checking (considering context when doing it), part of speech tagging and eventually even parsing started to work. Back then, people could still only analyze sentences with fewer than 40 words - the rest was often cut off. Then came more and more advanced machine learning models (decision trees, HMMs, CRFs), first a whole zoo, and then support vector regressors (SVM/SVR) ate everything else for breakfast. Then in machine learning a revival of neural networks happened, because better training algorithms were discovered, more data became available and cheap GPUs were suddenly available because kids needed them for computer games. This led to what some call the ¨deep learning revolution¨. Tasks like speech recognition where people for decades tried to squeeze out another half percent drop in error rate suddenly made huge jumps, improving quality by 35% - so jaws dropped. (But today's models like BERT still only can process 512 words of text.)

So it is understandable that people worry at several ends. To lose jobs, to render ¨NLP redundant¨. I think that is not merited. Deep neural models have their own set of problems, which need to be solved. In particular, lack of transparency and presence of different types of bias, but also the size and energy consumption. Another issue is that for many tasks, no much data is actually available. The big corps like Google/Meta etc. push the big ¨foundational¨ models because in the consumer space there is ample data available. But there are very important segments (notably in the professional space - applications for accountants, lawyers, journalists, pharmacologists - all of which I have conducted projects in/for), where training data can be constructed for a lot of money, but it will never reach the size of the set of today`s FB likes. There will always be a need for people who build bespoke systems or customize systems for particular use cases or languages, so my bet is things will stay fun and exciting.

Also note that "NLP" is a vast field that includes much more than just word based language models. The field of propositional (logical) semantics, which is currently disconnected from the so-called foundational models, is much more fascinating than, say, chatGPT if you ask me. The people there, linguist-logicians like Johan Bos identify laws that restrict what a sentence can mean, given its structure, and rules how to map from sentences like "The man gave the girl a rose" to their functor-argument structure - something like "give(man_0, rose_1)¨ - which models the "who did what to whom?". When such symbolic approaches are integrated with neural foundational models, there will be a much bigger breakthrough than what we are seeing today (mark my words!). Because these tools, for instance Lambda Discourse Representation Theory and friends, permit you to represent how the meaning of "man bites dog" is different from "dog bites man".

So whereas today`s models SEEM a bit intelligent, but are actually only sophisticated statistical parrots, the future will bring something more principled. Then the ¨ "hallucinations" of models will stop.

I am glad I am in the field of NLP - it has been getting more exciting every year since 1993, and the best time still lies ahead!

* Verbal translation, where accuracy is usually important enough to want to also have a human onboard since humans still have an easier time with certain social clues.

* High-culture translation, where there's a lot to personal choice and explaining it. GPT can give out many versions but can't yet sufficiently explain its reasoning, nor would its tastes necessarily match that of humans.

* Technical translations for manuals and such. This market will be under severe threat from GPTs, though for high-accuracy cases one would still want a human editor just in case.

All in all, GPT will contract the market, but many human translators will be fine. There's still areas where you'd still want a human, and deskilling isn't a bug threat - a human can decide to immerse and get experience directly, and many will still do so by necessity.

I've said it before and I'll say it again. This right here is the crux of the issue. The only way people get to eat is if we change the economic systems.

Capitalism supercharged by AI will lead to misery for almost everyone, with a few Musks, Bezoses and Thiels being our neofeudal overlords.

The only hope is a complete break in economic systems, towards a techno-utopian socialism. AI could free us from having to do work to survive and usher in a Star Trek-like vision of the future where people are free to pursue their passions for their own sake.

We're at a fork in the road. We need to make sure we take the right path.

What can possibly be the benefit of requiring this constraint?

Remove the idea that this is necessary and watch how much relaxation comes to the deliberation on this topic.

"Current economic systems" will simply have to yield. Along with states. This has been obvious for decades now. Deep breaths, everybody. :-)

My gut feeling is that AI is the 'social historic change' that will make UBI politically viable and a reality.

The problem is that we have created an economy where that is a bad thing.

Many of these jobs are cheap and easy to understand and quick to train in. These aren’t the kind of jobs people probably wanted, but they’ll be there.

Yes, they will be all called 'ai data labellers'.

For a long time, "People don't just want jobs, they want good jobs" was the slogan of industries that automated the boring stuff. Now AI is suddenly good at all the jobs people actually want and the only thing it can't do is self-improve. In an AI future, mediocre anything will not exist anymore.

Either you are brilliant enough to be sampling from 'out of distribution', or you're in the other 99 percent normies that follow the standard : "learn -> imitate -> internalize -> practice" cycle. That other 99% is now and eternally inferior to an AI.

I used to do a job that was eventually automated. We did the one and only thing the computer couldn’t do - again and again in a very mechanical fashion.

It was a shit job. You might get promoted to supervisor - but that was like being a supervisor at McDonalds.

Why not treat the job seriously? Why didn’t the company use it as a way to recruit talent? Why didn’t the workers unionize?

Because we all knew it would be automated anyway.

We were treated like robots, and we treated the org like it was run by robots.

There’s a huge shadow over the economy that treats most new jobs like shit jobs.

Maybe the very, very basic transcription/translation stuff might go away, but arguably this race to the bottom market was already being killed by google translate as bad as it is anyway.

In areas where quality is required (eg. localizing video games from japanese to english and vis versa) people would be (justifiably) fussy about poor localization quality even when the translation was being done by humans, so I have to imagine that people will continue to be fussy and there will still be significant demand for quality job done by people who aren't just straight translating text, but localizing text for a different audience from another culture.

The way it works, someone would have to produce more original training data in the first place. In long term, it is AI that has to keep up, not the other way around.

> wealth inequality

Microsoft, OpenAI run closed for-profit LLMs that are inherently only possible thanks to creative work of all the people that might stand to lose jobs now. Not only it should be clear where the driving force for rising wealth inequality is going to be going forward—these companies’ effectively scraping original works of living humans and repackaging them for profit should be in violation of intellectual property law, if it isn’t already. Perhaps more people should start adding two and two together.

Or you could still have a human in the loop, for example a CTO reviewing their AI developer’s code. But now you can pay one person instead of 10.

That said, these tools still need some domain knowledge. It’s not at the point yet where anyone off the street can use it to accomplish an unfamiliar task.

There’s an old story about someone charging $1,000 to fix a machine. They did it in 10 seconds and the client complained. The consultant said “I get paid for knowing how to fix it”.

The same might be true for knowing how to prompt AI.

Yeah, exactly this. The funny thing about economic upheavals and industrial revolutions is that while people might get reallocated eventually (say, 30 years), that doesn't provide any comfort to the people who are getting upheaved now.

Somewhere toward that goal we became overly fixated with money/wealth and the pursuit of endless profits. Meanwhile, companies continue to post record profits while downsizing - gee, I wonder why.

The reality is, we need to start to figure out and move toward a new way to run the world. I always semi-land in a kind of Star Trek: TNG 'creation socialism', or whatever it is they have as a means of structuring a society, where you have a replicator that can create almost anything for you. They also have 'intellect machines' that are often used to build things from just describing what you want, or in other situation to dig into engineering problems, etc. Things that we are now starting to do with our new GPT tech.

Putting all this another way – how are all these corporations going to continue to exist when there are no customers to buy their products, because there are no jobs left. Basically, money as we know it is going to go away, we won't be a society of limited resources anymore.

And again as a reminder, this isn't something I'm saying is 2-5 or even 50 years away (maybe?). What I'm saying is that set the timeline however you want, it is going to happen, and we need to start to plan for it now. Based on what's happening already, we're likely at the start of a 20 – 50 year transition towards a new form of society entirely.

Most likely society will reorganize around it and new professions that don't exist today will be created.

And we haven't even considered the positive impact of AI.

It could for example accelerate the process of drug manufacturing and genetic therapies that will considerably increase human's lifespan, triggering another form of social reorganization.

So it is really impossible to tell with certainty how the world will look like in 50 or 100 years from now.

I am optimistic and would like to think it will be a much better place, not necessarily perfect and just in every aspect, inequality and armed conflicts will probably continue to exist, but overall it will be better than it is today.

There was a lot of fear during the industrial revolution, prompting many intellectuals to have a very grim outlook of the future, particularly when it comes to social issues (yes you, Marx).

But ultimately, if you look at the data compiled decades later, such as GDP and life expectancy around the world, it is undeniable that social and economic changes resulting from the industrial revolution made the world a much better place.

Not a perfect one, but certainly much better than it was (at least for humans).

When I show ChatGPT to a lay person they don't really care.

When I show it to a professional copywriter they say that if they submitted this content to a client they would lose the client.

I'm reminded of when my son was learning to talk and everything he said seemed brilliant and coherent to me.

To any stranger it sounded like gibberish.

I think GPT is like my son, and all tech people are like excited parents.

Maybe the kid will learn to speak like an adult, but it can't yet

I think the problem is that money saved from making those jobs redundant is not going back to the society but few heads on top. That's the fundamental problem, company moves money from payroll to tech that replaces the people and any savings are entirely company's profits.

And then they go ahead on tax avoiding spree so increased profits don't even flow back to the society

There are hundreds of thousands of people making their living on translating, and now most of them need to find something else to survive now.

We are into a new reality and no need to lament the old one, because it is gone anyway

We keep committing seppuku by emerging technology to where hubris becomes a large scale financial or even a physical crash, and the smartest ones of us avoid the knee jerk reaction to quick over-adoption of new and unpredictable tech. The main problem is that there are so many forces pushing us towards adoption of whatever is packaged and marketed most heavily, and we need to incorporate better rollbacks and ways of gradually adopting new tech. Testing is also a quickly dying art, as seen with Twitter... This will be our downfall if not properly reigned in.

The people (self proclaimed tech leaders) who we exalt have already shown us that they are impulsive, and driven by greed, vanity, and ego. If we continue to let them act as "golden children", and continue to make them more and more wealthy, there will be no going back to fairness in our world. Tax them fairly, hold them accountable, stop letting them influence politics, stop electing rich people and people with conflicts of interest, and let everyone have a voice and equal opportunity to climb into responsibility based on their incremental successes... We're not doing any of that at all right now -- it greatly worries me.

This was true for Industrial Automation when machines automated physical work and humans could do knowledge work.

Now machines are on the verge of automating knowledge there would be no work left for humans.

Bad examples. Those are instances where you need human beings to provide interpretation of the context surrounding the translation/transcription, and where strict regulatory regimes are in place. Those are likely the last to be automated.

I guess the same intuition led to these new AI technologies...

Because for the other 20 percent it's plainly -not- good enough. It can't even produce an acceptable business letter in a resource-rich target language, for example. It just gets you "a good chunk of the way there."

And there's no evidence that either (1) throwing exponentially more data at the problem with see matching gains in accuracy or (2) this additional data will even be available.

The fact we got this far through brute force is just insanely telling. This is a natural phenomena we're stumbling upon, not something crafted by humans.

Also - fun fact, the Facebook Llama model that fits on a Raspberry Pi and is almost as good as GPT3? Also basically brute force. They just trained it a lot longer and it shrunk the model. Food for thought.

However, translation of more distant languages is pretty terrible. Vietnamese to English is something I use Google translate for everyday and it's a mess. I can usually guess what the intended meaning was but if you're translating a paragraph or more it won't even be able to translate the same important subject words consistently throughout. Throw in any kind of slang or abbreviations (which Vietnamese people use a lot when messaging each other) and it's completely lost.

Really, words, utterances by themselves, carry meaning. Language is just a structure for _us_, so to speak, that we agree on for ease of communication. I think this is why probabilistic models do so well: the ideas we all have are mostly similar, it really is about just mapping from one kind of word to another, or kind of phrase to another.

Feel free to respond, I’m most certainly out of my depth here.

I've done a lot of work in NLP and the times when computational linguistics has been useful is very rare. The only time I shipped something to production that used it was a classifier for documents that needed to evaluate them on a sentence by sentence basis for possible compliance issues. Computational linguistics was useful then because I could rewrite mulit-clause sentences into simpler single clause sentences which the classifier could get better accuracy on.

> And here was Google Translate being "good enough" for 80% of all use cases using a "dumb" statistic model that didn't even have a coherent concept of what a language is.

I assume you are aware if Frederick Jelinek quote "Every time I fire a linguist, the performance of the speech recognizer goes up"?[1]

That was in 1998. It's been pretty clear for a long time that computational linguistics can provide some tools to help us understand language but it is insufficiently reliable to use for unconstrained tasks.

[1] https://en.wikipedia.org/wiki/Frederick_Jelinek

At the margin, these are equivalent (Chinese room, and all). I wonder if humans also learn similarly then retroactively tell themselves they actually do know things instead of just containing experiences encoded in their neurons (and whether that is any different than a neural network encoding trained "knowledge" in its neurons, too). This is the semantics of epistemology, at the end of the day.

Which of course is a good thing to make sure many people get to keep their jobs.

Can you link to an example?

Since LLM’s can’t scope themselves to be strictly true or accurate, there are indeed good reasons, like liability for false claims and added traditional support burden from incorrect guidance.

Everybody is getting so far ahead of the horse with this stuff, but we’re just not there yet and don’t know for sure how far we’re going to get.

http://www.incompleteideas.net/IncIdeas/BitterLesson.html

That's an odd reason to want this.

  those projects were just PR so that c-levels could sell how they were preparing their company for a digital world

Anyway it's a symptom of the hype cycle - AI was the next electricity, but there were no actual products and nothing clear to do with it, just hire a bunch of kids to act like they were in a kaggle competition, or worse a bunch of PhDs to be under-utilized building scikit-learn models.

Now that there are (potentially) products coming along that at least bypass the low-level layer of ML, having an internal team makes no sense. Maybe the most logical thing that will happen is the pendulum will swing too far, and this bubble will consist more of businessy types using chatGPT without remotely understanding it or realizing it's just a computer program.

I wonder if this is a bad as everyone thinks. When a new technology arrives which is not completely understood, isn't the right approach to try to find some applications for it? Sure, most will fail, but some valid use cases will likely emerge.

I'm pretty sure almost all technologies at some point were solutions looking for a problem to solve. Examples include the internet, the computer and math.

You have to remember that when these sorts of things happen, the ones who get "taken down" in ways that actually affect their lives are invariably the ones who already have the least. The ones who "need" that takedown will be just fine, unless they've made incredibly stupid investment decisions.

Yeah having a whole big team create the internal baseline is not cost effective, but having at least one or two people work on something to actually know the vendor is worth their cost is important.

In fairness most PhD topics people work on these days, outside of the select few top research universities in the world, are obsolete before they begin. At least from what my friends in the field tell me.

Their value just went up tremendously, even if their PhD thesis got cancelled.

Easily millionaires waiting to happen.

---

edit: Can't respond to child comment due to rate limit, so editing instead.

> That is not how it works at all.

Speak for yourself. I'm hiring folks off 4chan, and they're kicking ass with pytorch and can digest and author papers just fine.

People stopped caring about software engineering and data science degrees in the late 2010's.

People will stop caring about AI/ML PhDs as soon as the challenge to hire talent hits - and it will hit this year.

Will this effect the job market (both academic and commercial) for these folks? It's very hard to say. Clearly lots of value will be generated by the new generation of models. There will be a lot of catchup and utilisation work where people will want to have models in house and with specific features that the hyperscale models don't have (for example constrained training sets). I'm wondering how many commercial illustrators have had their practices disrupted by Stable Diffusion? Will the same dynamics (what ever they are) apply for the use of LLM's?

And aren’t PhDs supposed have a theoretical underpinning?

To be honest, that's a sh*tty PhD to begin with, it doesn't bring anything good and really worthwhile to the world, quite the contrary.

We could not "just use ChatGPT" because the answers have to be correct, or students could take legal action against the university in situations where they'd give an answer in an exam that the chatbot gave them and that might be wrong. It's exciting for sure to compare e.g. ChaptGPT with other approaches, for a set of controlled benchmark questions such as:

  -- snip --
  Q: What is the A* algorithm?

  A: The A* algorithm is a popular heuristic search algorithm used in pathfinding and graph traversal. It is an extension of Dijkstra's algorithm, but uses a heuristic function to guide the search towards the goal node, which can make it more efficient than Dijkstra's algorithm in certain situations.
  The algorithm works by maintaining a priority queue of nodes to be explored, where the priority of each node is the sum of its actual cost from the starting node (g-score) and an estimate of the cost to reach the goal node (h-score), based on a heuristic function. The heuristic function must be admissible, meaning it never overestimates the actual cost to the goal node.
  At each step, the algorithm selects the node with the lowest priority from the queue and expands it, generating its neighboring nodes and computing their priority scores. The algorithm then adds these nodes to the queue if they have not been explored before or if a better path to them has been found.
  If the goal node is reached, the algorithm terminates and returns the path from the starting node to the goal node. If the queue becomes empty before the goal node is reached, then there is no path from the starting node to the goal node.
  Overall, the A* algorithm is a powerful and widely used algorithm that has many applications in fields such as robotics, video games, and logistics.
  -- snip --

This is part of what the original UBI concept was about.

If this doesn't happen, yes, there will likely be violence until it is fixed.

The other view is that many technologies that were supposed to reduce work actually net added work, because now more sophisticated tasks could be done by the humans, so the net was similar to the highway paradox where more and wider highways breed more traffic by induced demand.

Where would this demand come from? IDK, but at least initially, these LLMs make such massive errors that keeping a lid on the now-hyper-industrial-scale bullshit[0] spewed by these machines will make many more full time jobs.

Seriously, just today I was amazed at how the GPT model tried to not only BS me with completely fabricated author names for an article that I had it summarize, but it repeatedly did so even after being successively prompted more and more specifically to where it could find the actual author (hint: right after the byline starting with the word "Author". It just keep apologizing and then doubling down on more fantastic lies, as if it were very motivated to hide the truth (I know it's not, that's just how fantablous it was).

[0] Bullshit being defined as speech or writing telling a good tale but with zero regard to the truth or falsehood of any part of it — with no malice but nonetheless a salad of truth and lies.

As mentioned elsewhere this is not the first technological disruption in the economy. The change from heavy industry to a service industry didn't go well, hopefully it's possible to take learnings from this and do it right this time.

I guess we have to get used to software redefining the meaning of words. It was kind of funny when that happened regarding Google Maps / neighborhood names, but with LLMs it's a different ballgame.

https://github.com/ogkalu2/Human-parity-on-machine-translati...

But the interesting thing IMHO is the nature of the mistakes ChatGPT makes... often they're quite elementary mistakes (e.g., the occasional subject-adjective word order) while it gets the big picture right. Whereas DeepL is sometimes the reverse. ChatGPT also has the advantage of being able to tailor its output to a particular context, e.g., it can tailor legal translations to terms used in Canadian law rather than French law. However, for longer texts, I've noticed that ChatGPT will sometimes omit small parts of the source text from the translation, which is unfortunate.

I have a colleague who says that the Mandarin translations done by ChatGPT are an order of magnitude better than DeepL though, which is interesting.

https://youtu.be/Bf-dbS9CcRU

With a PhD in the domain, I consider myself pretty good at (a subset of) distributed programming. But these days, when companies hire for distributed programming, they seem to want developers who know a specific set of tools and APIs. I'm more suited at reimplementing them for scratch.

That seems to be running directly opposite of the current trend of admin assistant jobs requiring 2 years specialized admin assistant diplomas. Tech (and I would guess the world of the business MBA) is a unique space where people are learning and changing so quickly, but for a lot of those outside the bubble things seem to be calcifying and requiring more and more training at the expensive of the worker.

Why the hell stay in in academia? This is clearly the next technological wave, and you shouldn't sleep on it. Especially when you're so well positioned to take advantage of your experience. You can make $500,000/yr (maybe more with all the new startups and options) and be on the bleeding edge.

If you want to go back to academia later, you can comfortably do so. Most don't, but that doesn't mean it isn't an option.