Choose Your Weapon: Survival Strategies for Depressed AI Academics
arxiv.org
arxiv.org
OpenAI is not doing science. They are building a big shiny thing, showing it off, keeping it closed, and making money off of it. That is not part of the scientific process.
It is as if, every time SpaceX launched a rocket a little bit higher, every aerospace department ooh'd and aah'd and threw up their hands: "we don't have the resources to build a bigger rocket; how can we do science?"
Unfortunately, the research field of AI is diseased: it places way too much value on showing off big shiny things above progress in scientific understanding. But there is a ton of progress available to be made by small teams on small budgets. There is so much we don't understand about neural networks alone, even small ones.
That's exactly OP's point
There's also lots of room in applying AI techniques to other areas of science (again, outside of professorships)
Looking for it in UK job boards doesn't seem to bring many relevant results. What are other ways to find these roles?
The idea of working in a lab helping scientists, if not economically rewarding, may feel rewarding in terms of helping science to progress
What? OpenAI is absolutely conducting science amongst themselves. They are certainly engaging in the hypothesis -> test -> result scientific method, which is science in its purest form. Sharing the secret sauce is what is done or expected in academia, but that doesn’t mean that OpenAI isn’t conducting science behind their own walls. You’re conflating institutional academia with the scientific method.
If I go home and test the boiling point of water in my garage and record the result privately without publishing a paper, am I not conducting science? What nonsense.
Edit: I mean, c'mon - the final step to the scientific method is "communicate your results". I'm surprised this requires explaining.
Science is knowledge. The etymology of science is to split stuff and understand how it work. But from late 14c in the more specific sense of "collective human knowledge".
"Private" science is not science. It's some esoteric stuff.
OpenAI, even without saying how the model works, has been hugely influential in science.
If science happens behind closed doors but we witness what it makes possible, that influence alone can inspire a lot more effort and discovery with or without the data that lead to the accomplishment.
I think that moment has been profoundly influential due to ChatGPT alone.
Whether or not it's a good thing that it's largely private is very difficult to say. I don't feel great about it, but I also don't think I know enough to have strong opinions yet.
"Public" science is just a special case of "restricted" science, such that the set of people the knowledge is restricted to is equal to the set of all humans.
An essential part of doing science is sharing results and waiting for others to challenge, expand, or falsify them. If you skip this step, you're not truly contributing to knowledge, although you may still create new knowledge in other ways... but not all new knowledge is necessarily scientific, it can be private, esoteric, commercial knowledge.
Take the example of Ida, a 47-million-year-old lemur fossil known as Darwinius masillae. https://en.wikipedia.org/wiki/Darwinius it was "discovered" and hailed as a transitional species bridging primates and mammals. Really a huge contribution to "science".
But here's the catch - Ida was actually found in 1983 and held by a private collector for 25 years, who didn't realize its significance. So we had no idea of this "discovery". It wasn't science, it was a good on a market.
OpenAI is the same. We might enjoy to toy with the technology. But I don't find it contributed to science, yet
Let's say that there was a highly advanced civilization that existed way before the current civilized society. They performed scientific experiments and gained a lot of insight, but a catastrophical event erased them from the face of Earth together with all their knowledge.
Did that civilization engage in science?
If you think that they did, indeed, engage in science, then what is the difference between them and any other closed community that performs scientific experiments?
Or, if you think that they didn't, in fact, engage in science, then does that mean that science is only science if you and I, today, can gain access to the knowledge derived from the experiments? Therefore, science is defined by its utility to us?
This thought experiment seems to introduce a huge variable which eliminates what I think might be a key factor in what many readers believe to be the practice of science.
> what is the difference between them and any other closed community that performs scientific experiments?
That difference here, and the key factor, would be intent. The catastrophic event eliminates any meaning behind what they accomplished or practiced in regards to why they practiced science in the first place.
Practicing it behind closed doors may appear less like science for science's sake and more like, say, an economic pursuit that coincidentally involves the scientific method.
That could be why the conversation appears so split at the moment. It's hard to claim science is about intent or a long-term purpose that, were it removed, it would bar a scientific activity from actually being Science proper. That's a very subjective thing.
But if they didn't suffer a catastrophic event, we would call their activities "science"? So, because they did suffer a catastrophic event, what they did is not science (or is not relevant, since you avoided answering the question directly)?
And if our current society was destroyed tomorrow in a catastrophic event along with all our science, and a new society arisen from the ashes that performed scientific experiments, would that mean that what we did wasn't science? Or that what the new society is doing isn't science?
I know I'm streching things here, but words (should?) have meanings in all possible scenarios, and thought experiments are used to decide meanings in such edge cases.
> Practicing it behind closed doors may appear less like science for science's sake and more like, say, an economic pursuit that coincidentally involves the scientific method.
Does "not doing science for science's sake" include only monetary motivation? Would it also mean that someone who primarily wants to achieve status and fame as a scientist also isn't doing real science? Or are they exempt from the rule, since they would most likely release their knowledge into the public (which brings us back to the "science is only science if the current society benefits from it")?
Take, for instance, the Greco-Roman mix for concrete. They produced that scientifically, repeating experiments with their volcanic ash-filled cements to make the best they could, and shared those results across their territory. While that knowledge was lost for centuries, it was still science.
However, a cement company producing a better product and not sharing the production is not science. It’s an economic pursuit. Its goal is not to help everyone to produce a better product, but to produce a better product and hide the results. Even if it uses most of the scientific method, if it doesn’t share its result its not science.
And no, monetary stake isn’t the only reason someone may not be doing science properly. A major one we see time and time again is the desire for science-based renown.
For instance, Einstein is renowned for being one of the smartest men alive, who shaped a lot of our modern understanding of the universe. He may ultimately be wrong, but we have yet to prove that and in fact continue to prove he was decades ahead of his time.
Contrary to Einstein though is Edison, while he did do quite a bit of real science, he was less focused on actually advancing our knowledge and more focused on his own image. For instance, he invented the electric chair solely to try and slander Tesla’s AC, and tortured animals in public to try and breed fear.
Additionally, I will throw my hat in the ring and say most of modern academia doesn’t pursue science, as they push for ever-restricting holds on scientific papers in the pursuit of financial gain. While simultaneously depriving the actual researchers any of that gain, money which could surely help raise a lot of institutions into doing more effective research. Instead, they profit off of and restrict the free flow of knowledge to line their pockets, disabling millions of researchers worldwide from accessing it and imposing harsh prices on non-academics who wish to learn.
In the end, when it comes to what is and isn’t science, I will posit a question. If you never heard about it, and can’t reproduce it yourself/watch someone reproduce it, did it really happen? Or has “science” evolved from the pursuit of knowledge to a faith? If we blindly trust the people in power that they are right, and punish those who question their assertions, it can only end in destruction.
The scientific (and then, industrial) revolution didn't begin until researchers started sharing their results in open literature form, or at conferences, or within bodies like Britain's Royal Institution. Dissemination of knowledge is absolutely part of science.
Similarly, a fundamental necessity in science is independent replication of results, and that's definitely not possible with closed secretive research done in the alchemical mode.
There are many examples, such as the secretive public-private DOE FutureGen program aimed at "zero-emission coal-to-hydrogen plants" which wasted about ten billion dollars over twenty years with nothing to show for it (and whose data is still not available), the Trofim Lysenko plant breeding program in the Soviet Union which wrecked Soviet agriculture for decades (and falls into your latter category) and so on.
Another example could be the Challenger Space Shuttle disaster, although that's more an engineering failure than a science failure, but it had similar underlying causes, i.e. managers pushing engineers to go along with a flawed protocol.
You're right that "OpenAI is not doing science" was hyperbolic, but I tried to clarify the aspect of science that I'm referring to.
For example, at minimum psychology, epidemiology and climatology have this problem. In climatology attempts to get the source code of models and data underlying papers has led to some of the biggest dramas in the field e.g. the Climategate hacker appears to have been motivated by the closedness of the field (similar to the llama leaker).
Given that society is clearly not ready to reclassify large parts of academic work as non-scientific, it seems like this definition cannot work even on its own terms.
OpenAI provides me with a real, useful tool, something that directly improves mine and many others' lives. Your papers do not have that effect on my life.
> OpenAI provides me with a real, useful tool, something that directly improves mine and many others' lives. Your papers do not have that effect on my life.
Of course, OpenAI's work is built directly on top of publicly-funded research papers of the past 5, 10, 50 years.
This applies pretty much to ..euh every field. The wow factor attract investors and money. The way the current economy is structured, it incentives these tactics since these individuals will be able to draw on significant funds and will have a comparative advantage even if they have a worse return ratio.
And then it'll pass and a new thing will kick again. Investors have no memory, and I am talking about institutional ones. The supposedly sophisticated folks.
These CS academics need to understand that this is what it feels like to be in other fields like physics or bio where you can’t do jack unless you have costly equipment. This is what people in developing countries deal with all the time.
And people will build the next GPT or whatever and even that’ll get boring. The people in big tech have to pivot and do whatever the economy demands. Whereas the academics can go back to writing papers like nothing changed.
The trick is always to offset the cost of inference with larger cost of training. You don't apply the Chinchilla scaling law, that's for academics and people who don't have to pay for inference. You pretrain the model 10x longer (like 1T tokens for LLaMA) to make it the best you can fit into an A100 or 4090 quantised to 4 or 3 bits. So everyone can have AI assistants running on their own toys.
This happened in 10 years (training cost of largest model according to the batshit crazy god AI article in the FT).
On the one hand compute power increased by about 100x and then some lunatics came and spent 10000x on it...
This was my thought too. It cost approximately $4.75 billion to build the Large Hadron Collider at CERN. My educational background is in physics, but I would not be the least bit upset if a big chunk of public research funding was shifted from physics to CS/ML/AI. It’s obviously far more important to the future of humanity than finding the Higgs boson.
Not to mention that there is zero appetite from undergrads or postgrads to get into the nitty-gritty of it. To learn CNNs at the deep-dive level you need calculus, at least differentiation and integration. Calculus or even pre-calculus doesn't form part of the degree programme for most compsci BScs any more, because it is 'too hard'.
The way most students 'learn' AI is to use a method out of a Python library with near-zero understanding of how it works, and regurgitate it for an assessment.
Professorial research staff in most UK universities are light-years from AI within industry, and there's no clear path to that gap tightening, especially while universities are being run like second-rate consulting houses (don't get me started on THAT).
anyway i think your statement that industry is light years away from unis is just misleading. i think the two are trying to answer different questions: 1. how can i achieve a "somewhat" decent chatbot that gets me rich albeit not even knowing what it does [industry in case you wondered] 2. try to understand, quantify and measure how well a model works, is it stable? does it converge if we have small datasets? and so on so forth.
just my two cents, to conclude i think a good analogy to the current climate is the 700-800s with electromagnetism: plenty of people discovered "empirical" laws but didn't understand really the phenomenon.
Sounds dead on. Do these large """language""" models actually even implement any concepts from linguistics? Or is the entire "language" part of the model merely derived from the fact that it's inherently part of the training data?
I don't fault Chomsky at all for being fed up with the hype here.
The entire field is also glossing over the fact that other languages which aren't English exist.
GP here is, IMO, confusing what the corporations want (1), with what corporate R&D people want (2). As long as the corps see good ROI on throwing infinite money at their AI R&D departments, then those corporate researchers are better positioned and better equipped to do actual, solid science, than academia ever can be. This has happened many times before, including in this industry. Research is best done by well-funded teams of smart people left to do whatever they fancy. When those conditions arise, progress happens, and it doesn't matter whether it's the government or industry that creates them.
(Conversely, the best hope for academia to become relevant again is that corporations lose interest in this research, and defund their departments. This could happen if e.g. transformers end up being a dead end, or compute suddenly becomes very expensive.)
> Do these large """language""" models actually even implement any concepts from linguistics? Or is the entire "language" part of the model merely derived from the fact that it's inherently part of the training data?
The latter. And guess what, they're not trying to solve the issue of linguistics. They started as tools to generate human-sounding text, but in the process of just throwing more data and compute at them, they not only got better, but started to acquire something resembling concept-level understanding.
It turns out that surprisingly many aspects of thinking seem to reduce well to proximity search in a vector space, if that space is high-dimensional enough. This result is both surprising and impactful well beyond the field of AI. It's arguably the first potential path we identified that the evolution could take to gradually random-walk itself from amoeabas to human brains.
btw, in other languages i guess it is decent although it depends on which language, at least gpt4.
What do you mean? Processing a CNN layer takes an amount of time that does not depend on the input data, only the input/output sizes. Fourier transform is just a change of basis. Why should anything speed up?
This seems to be done in some cases. I guess it isn't done more widely because the "standard" convolution kernels are very small and the performance would actually be worse?
I think your complexity argument is correct for N=pixels=kernel size. But typically, pixels>>kernel size.
Disclosure: I work at Arm optimising open source ML frameworks. Opinions are my own.
In the US I’ve never seen a BS in computer science that didn’t require calculus. I can’t speak for the UK, but it would surprise me that what you say is true.
Oh the poor dears, imagine needing schoolboy maths to do science.
That's interesting, could you please elaborate?
You can easily see a lot of people who work at OpenAI on LinkedIn. University AI labs are always left out because non-commercial products just aren’t as noticeable.
I highly doubt that many of these academics would struggle being amount candidates at DeepMind, OpenAI and FAIR.
Prestige is very subjective, which makes it also very broad thankfully.
But there's a different way to look at it. Because industry went off and solved the now-"boring" problem of building ever-more-powerful computer processors, the people left behind in academia got to invent machine learning, public-key cryptography, modern coding theory, distributed systems... and so on. What makes academic research valuable is not the freedom of "I get to plan my own day", but the freedom of "I get to work on the weird problems that industry doesn't even realize are problems."
In machine learning we already had back-propagation in the 1970, but had to wait another 40 years for Intel and Nvidia to create ever more powerful processors to tackle useful problems.
So I get how they feel - one of the coolest problems ever instantly went from being something anyone could hope to contribute to, to something almost no one can. You can't match the compute to do the things OpenAI & Google & friends do. And if you happen to stumble on something related that can be explored without access to obscene amounts of capital, guess what, the corporate research teams will notice it, and they can do it better than you, and then they can the idea much further than you ever could.
I'll counter that. In the end we need AI that can do training AND inference on edge devices out in the real world. A good (and possibly profitable) example would be robotic pets that can learn (even to understand words) and interact with their owners like real animals, but don't need to go to the vet or eat and poop. Big companies relying on huge compute resources are not even aiming at this type of thing. They're too busy using their "scale" to even bother looking at smaller but interesting methods or solutions.
That's of course if, by this point, we aren't in a middle of a futile scramble to avoid getting extincted by ChatGPT-7 that someone left in self-play mode and forgot to turn off before going on vacation.
Point being, general AI is general. Even at extreme expenditure of resources, the closer the corporations get to it, the more problems they can put it to - including, eventually, the problem of optimizing itself. Already today people are using current-gen models to assist in developing next-gen models; this trend will only continue, until at some point you'll be able to let the model self-improve, mostly unsupervised. I imagine the compute costs per AI value delivered will drop like a stone then.
From what I can tell, the team at OpenAI (following after Google/Deepmind etc.) are simply mashing the pedal to the floor to get bigger and better models from their existing techniques, and then tuning the resulting black box to make it produce more "useful" answers. And that's fine! That's precisely what an industry lab is expected to do: they have the resources to do the training and the need to get products in front of paying customers as quickly as possible to justify it. And frankly with top AI engineers getting paid millions of dollars and Google/Meta tight behind you, emphasizing results is the most viable strategy. If "turn the needle on the box to the right" is giving you good answers, why would you waste a $1-$5m-salary engineer on academic questions like "why does the box do that?"
And yet, asking questions like "why does the box do that?" is the reason technology didn't stop at the steam engine. I suspect that finding the answer to those questions won't immediately sell enterprise licenses, but will be very important. And the answers will probably fall to someone who's making $30k/year in a graduate program.
ETA: Of course, it may turn out that "turn the needle on the box to the right" is enough to obsolete all human researchers, in which case I'll be wrong about this. But it'll hardly matter in that case ;)
I'm only an amateur in the field, so my uneducated high-level understanding is that, in a sufficiently high-dimensional latent space, there's more than enough dimensions to assign to any single semantic relationship people ever thought of, which is what the training process effectively does, which reduces an important part of thinking - working with concepts and their relationships - entirely to vector adjacency search.
I'm only beginning to study the details, and I don't know how much of specific understanding of this exists, but at the very least this high-level model explains why scaling makes qualitative difference here.
Now, I agree they have strong commercial incentives to push their models as far as possible as fast as possible, but honestly, if I were a researcher working on these models, even if I was somehow unconcerned with any kind of commercial viability and had access to more compute, I'd absolutely keep scaling those models up and up, all the way until I hit the limit of available compute, or the models stop qualitatively improving with scale.
Basically, there's no reason[0] to stop now and try to fully comprehend how GPT-2 works, when GPT-3 was a qualitative jump, and GPT-4 even more so, and GPT-5 is around the corner, and GPT-6 might be a year away from now. All those steps yield important new insights into how the whole architecture works, and if at some point the scaling breaks, that would be even more important knowledge to have. And this doesn't even take into account the fact that, starting with GPT-3, those models are increasingly useful in accelerating both research and scaling alike.
----
[0] - Except, of course, that if transformer models are the road to generic AI, then we'll just blindly race straight into a point of no return.
I would disagree with this description. An "emergent property of large networks" would be something that just appears when you wire together a large network.
To get intelligent behavior, it's not sufficient to wire together a large neural network. You also need to use an optimizer to train it on a large data set.
I cannot see anything that remotely makes these two contexts architecturally, practically, or ethically similar, except in the narrow sense of query responses that might have been mistakenly thought of as defining sentience in earlier decades.
This leads to believing that GPT has solved many of these issues.
Of course that's not a criticism of the ML achievement, except that everyone and their mother has started making some extreme arguments about the thing.
> language enables intelligence
In his research, he argues that it’s the other way around; we have an innate intelligence for acquiring language.
To put it another way, language has made us more knowledgeable (because it’s a great mechanism for building and transferring context), but intelligence or aptitude for language was present before we developed languages.
Does this align with how your views were reshaped post LLMs?
> language enables intelligence
was just a poor choice of words here. i understood GP as "the structure of a language influences its speakers' worldview or cognition, and thus people's perceptions are relative to their spoken language"[0]
My argument is just that:
- Sentience is variable. We might regard primitive organisms as biological machines. As some of these machines have slowly evolved to become more complex, sentience/consciousness has emerged, effectively by 'redesigning' and scaling the same hardware. It is an emergent property. At the same time this has been happening, other aspects of what we call intelligence have been developing.
- Artificial networks being scaled, are also displaying emergent behaviours, some of which overlap with aspects of intelligence. This does not imply they are sentient, but it does strongly support some previous assumptions that sentience, like intelligence, is emergent and hence there will be no better answer coming.
There's very clear definitions for sentience & sapience.
The only confusion is that laypeople tend to conflate sentience & sapience.
A sentient AI is not necessarily a sapient one, by definition.
The skeptics are really not making any claims are they? If not then there's no burden of proof on them, is there?
Whoever says or implies that ChatBots are or are "becoming" "sentient" are the ones making a claim. They should prove it if they make or imply such a claim.
Skeptics are rightly skeptical, until they see a proof with clear definitions of terms from those who are saying or implying that ChatBots are 'approaching sentience".
There are no two "sides" making claims about AI. There is one side, making claims such as "AI can be/is/may be sentient".
If somebody makes such claims they have the "burden of proof" because they are the ones making the claim. It's not a moral obligation to prove it. "Burden of proof" simply means if you don't provide a proof, nobody (I would hope) will believe you.
Note the skeptics are not making any counter-claims, they are simply saying if you make such fantastic claims about current AI, please show us the proof, else we rationally cannot trust your judgment.
Just as a made up discription lets say this
Them: "If $object does A, B, and C, then it is a bajork"
Me "I have $object that does A, B, and C, it is a bajork"
Them: "No, it also doesn't do D"
Me: "That's goalpost moving"
Them: "Well, it doesn't do B in the way we expected, so it doesn't really do B"
Me: "Maybe we poorly defined what B was in the first place, that doesn't mean we didn't do B, it seems to mean no one understands what B is in the first place"
Like if we have models that are plainly more intelligent and "emotionally responsive" than say, frogs or even dogs, do we still do we still take the approach of "If you can't prove its sentient, do whatever you want to it". Which of of course, we can't even prove dogs have consciousness.
OP said:
> plainly programmed algorithmic systems
Compare that to a human brain which is not a linear, 1D algorithm.
The reason recent results are significant in this discussion is because:
- Larger networks are already displaying emergent behaviours that couldn't have been predicted beforehand. It doesn't have to be 'human-like' intelligence for that to be significant.
- We're getting close to a point where our models will start exhibiting something of the order of the same level of intelligence of primitive creatures.
So the wide-eyed hysteria you're seeing is really just new data points confirming what many of us always assumed would be the case.
This explains nothing. Stating it's an emergent phenomenon gives us just as much information as saying it's magic. Arguably even less, because it sounds like an explanation, whereas "magic" doesn't pretend to explain anything.
> displaying emergent behaviours that couldn't have been predicted beforehand.
Same problem here. You say "couldn't have been predicted". Which is only true if you explain it away with emergence. The behaviours weren't predicted, because we don't understand how these networks work. Saying they couldn't have been predicted implies that we can't understand them. Which is a very bold statement and I'm fairly certain that it's not true.
I don't think that's the case. Of course at one level we can understand how GPT4 works, we have a working implementation of it after all. But that doesn't help us predict how it will behave, because its properties are emergent, and have to be studied separately at that level. That's not saying it's magic.
Reality can be unsatisfying because it does not owe us explanations.
I mean, you experienced the reality of consciousness while talking about the last digit of 22/7.
If there is anything in our world that needs explaining it’s consciousness.
Reality doesn’t owe us an explanation of consciousness (obviously). We owe it to ourselves though.
That is not at all quite plain. Respectable philosophical views include the proposition it is at least as fundamental as space or time.
Dismissing serious ideas with thousands of years of philosophical history behind them as "magic thinking" is either a failure of imagination or education; you can rectify the latter by reading up on the subject.
Edit: I wonder if there are studies that have determined whether or not new born infants are self aware or does it take time for it to emerge after the brain has been trained on its environment for a while
Isn't most ethical behavior driven by self-interest? We want to follow general principles that promote respecting those who are similar to us in the hope that others follow the same principles when interacting with us.
AI is too dissimilar for us to worry about whether its "mistreatment" would make it more likely for others to start mistreating ourselves.
- Twin Peaks, 1993
On the other hand, it is fascinating how much interesting work and theories in the field will be retired, at least for a long time (I allow for a likelihood that some of that could return eventually as better, more concise generalizations). Ten years ago Chomsky and various GOFAI-related stuff could still sound respectable and somewhat plausible. Plenty of people were/are wedded to the concept that you can build machine intelligence as an abstract and intellectually stimulating gentleman pursuit. It was a comfy position in a way. Not surprised about the resistance they are mounting now, though I think academic inertia will allow them to exist for quite some time. They can just move to doing some kind of philosophy also.
I would also disagree that ChatGPT constitutes a self. If anything, it constitutes a normal distribution over possible selves, where you get a random pick at the start of each conversation, and confined to the length of its token buffer.
I find interacting with GPT4 feels very different to interacting with either humans or animals.
These machines are merely thinking machines.
A rock is also more than it's thoughts, and retains some degree of consciousness. Imagining concepts like souls as a sort of litmus test for consciousness only serves to reinforce a thought-based comprehension.
In other words, when I control my thoughts, what is doing the controlling?
(see George Gurdjieff's philosophy which speaks to this issue of the `mechanical vs the conscious`, but please don't run off and join a crazy cult and ruin your life).
This seems like a change of topic. And vague.
But since you bring it up, I don't see anything threatening about AI.
First, if something is true, then, as a matter of principle, we have an obligation to believe it. Thus, the only threat the truth can pose is to falsehood, which is not a threat, but liberation. The truth shall set you free. Now, it may be uncomfortable is the truth is at odds with what you want to believe (a phenomenon very much present in the linguistic engineering we're seeing in the political sphere w.r.t. political correctness and its dishonest and obfuscating euphemisms), but unpleasantness isn't a threat.
Second, if we replace "threaten" with "challenge", then we might as what beliefs does AI actually challenge? How does it challenge this "mystery of the self"? That we can simulate human discourse or behavior with greater sophistication? That machine automation is becoming more sophisticated? I see no mystery where AI per se is concerned, only the mystification of those who wish attribute to it properties it does not possess, or those with intellectually superficial metaphysical commitments, like mechanistic materialism. AI cannot abstract from particulars, it has no true capacity for intentionality, to name two features central to intelligence. All of what the undiscerning and those given oven to fanciful notions see in AI is a projection.
And I do not think the classical[0] and traditional[1] thinkers viewed intelligence in the obfuscating manner that moderns infected by reductive, mechanistic materialism seem to.
[1] https://edwardfeser.blogspot.com/2019/03/artificial-intellig...
The race isn't to build the biggest model, the race is to build the model that uses data the most efficiently and produces the best results when constrained to parameter counters that can fit on consumer hardware.
AI's a big deal, but there's no reason to suggest that this makes any inroads toward unraveling self or consciousness significantly beyond what we get from books, movies, or perhaps more recently, video games.
>This goes a long way to explain the resentment that many AI researchers in academia feel towards these companies. Healthy competition from your peers is one thing, but competition from someone that has so much resources that they can easily do things you could never, no matter how good your ideas are, is another thing.
This sounds like teenagers whining they can't all be popular. Science isn't a competition to be won so that you can get praise and attention. Science exists to discover things that then hopefully are useful to people. If someone that isn't you discovers something useful that isn't bad because then you can't discover it. It is good because that is a problem solved.
The resources requirements FOR SOME SPECIFIC PROBLEMS have gone up to a ridiculous degree, but there are plenty of problems left to solve. In fact a new one that is at least as important has been created: Replicate current results with less hardware/parameters/whatever.
The difference between having GPT4 as a slow and costly service that requires network calls vs having it locally with almost no cost will be a huge achievement. Stop sulking and get to work!
Conversely business seems to be a competition which wants exactly that, no matter what the long term consequences are.
Lead in gasoline makes the engine stop knocking, great. Problem solved, nothing to worry about, right?Don't worry about funding scientists to look into it, it's all going to be ok.
I was also considering building a gaming pc to also be able to play around with ai / ml, only, I'm not even sure what one would be able to run locally anymore even with a 4090?
Overall? I feel you. It makes me worried for the new grads today. I don't really know if this old dog is up for new tricks.
There you go. Just fix this idiotic "publish (NeurIPS) or perish" attitude already!
Academia still plays an important training ground role, and it could shift focus to those areas that could be profitable but require coordination that is generally not feasible among certain institutions.
1. Really a blog post disguised as a paper.
2. Silly.
3. Excellent.
Properly, for that matter.