Big tech and the pursuit of AI dominance
economist.com
economist.com
We have been corrupted by words like that.
Why should we care about dominance?
We should care about collaboration and people’s well being.
Barring a catastrophe, AI is not going to stop.
At the current rate, by next month we will have some previously-never-seen model/algorithm/chip/tech that will run locally on our phones and computers. At that point is game over for the big server-model companies.
After that… well, pretty much all companies will be at stake.
There’s a much bigger revolution happening right now than just the tech industry.
We will be forced to ask ourselves what it means to be human.
No company is going to dominate AI, AI is going to dominate AI.
It seems likely there will be AIs specialised in taxes or medicine, people/systems will consult multiple AIs to smooth out errors and look for consensus etc. In short, I agree with OP that we will probably end up with something that looks more like an ecosystem than a hegemony.
And aren't we finding out that the amount of parameters seems to matter less than we used to believe?
This comment is so wildly inaccurate and hype, and yet so common in the GPT craze on HN
It is not happening on your phone and computers in the near term for 2 reasons:
1. Entropy, the models are LARGER, NOT SMALLER. The amazing performance come from the fact that they are BIG. This is opposite of Moore's law, which itself is ending.
2. Companies want to make money on their models, vending a client is clearly not the direction that companies are going. In fact greedy companies wanted everything to happen on the server end even without any real customer need, e.g. Jira.
Check it out. They are claiming 100x faster training and inference with smaller models, performing almost with the same accuracy as GPT.
People are already running models on their phones. Some people have been posting here and on Twitter how they can run Alpaca (Llama + GPT fine tuning), on their Pixel phones.
2. The cat is out of the bag. People are creating and releasing stuff faster than companies can keep up with.
Have you seen what ChatGPT is already able to do with plug-ins? You can give it control of a Linux machine and tell it “create a basic CRUD app and run it on a docker container”, and it will do it.
That means soon AI is going to be training itself, or at least with minimum input/direction from humans.
Here’s a good write up about that plug-in and an overview of a lot of the things you can do: https://andrewmayneblog.wordpress.com/2023/03/23/chatgpt-cod...
If you don’t have access to plug-ins yet, you could do it yourself by using the GPT api, writing a prompt that tells GPT they are interacting with a terminal and to only output terminal commands, then pass the output to a terminal, and passing back the output to GPT in a loop.
I can’t find the exact link to the docker app video/article.
Lately it’s really hard to keep up. The search feature here on HN kinda sucks on mobile and Google is hopelessly behind with how fast things are coming out.
Also what is “close”? The benchmarks I saw showed around 80% accuracy compared to GPT.
And maybe MS found the perfect arrangement giving them an ability to adjust how closely to associate themself with the tech according to need.
But is this all necessarily a good thing? OpenAI gave MS access to GPT and MS clearly wants to use GPT to destroy Google, so Google now has respond aggressively, and now we have exactly the AI arms race that many predicted would lead to the worst outcomes.
I mean not a single “big tech” company is offering any LLM products not powered by Open AI. It’s been almost 3 years since GPT-3 was released. I’m not sure they can catch up at this point. And I think the GPs point was a good one: big tech threw billions and billions at personal assistants. Nothing innovative was delivered.
Anything that can be provided by a micro or small sized company can be provided by billion dollar corporations too.
Maybe size brings a drawback, in the same way a mega ship finds it difficult to turn around without a day's notice — but these companies can and will.
Microsoft once betted all chips on the internet being fad; look at them now.
I thought it was extremely adept change of referring to it as machine intelligence instead of artificial. There's nothing artificial about this intelligence, merely different. I think this will probably be increasingly more correct moving forward with additional advances.
But then again, I don’t think humans are either.
Why would this be? Because AI allows you to copy your competitor in ways that was never possible before. It essentially becomes a skill and technology replicator. For example, look how Alpaca was able to somewhat replicate the multi million dollar models with $600 dollar cost.
I’d really like to start acquiring them. I know about The Pile and Common Crawl.
Are their any resources for individuals, that happen to have large amounts of storage, in regards to acquiring the best datasets being used?
I did lie a bit; OpenAI does have private datasets. For example, GPT-3 was trained on Internet data, but InstructGPT/ChatGPT was further finetuned through their own dataset of question/answer pairs. This was augmented in a really clever way: the initial set of valid completions was written by humans, but then OpenAI made more AI models to augment their training data before finetuning GPT-3 on it.
Even then, there's techniques for taking the output of one large language model and using it to finetune another. So just having a publicly-available AI chatbot is enough to get it to generate a dataset that you can then train another AI chatbot on.
I personally have been scraping Wikimedia Commons for public-domain imagery to train an art generator bot on; but this is mostly because I want an art generator that isn't full of pending copyright infringement lawsuits.
What I can’t get really pinned down is how much storage one needs to have your own copy of all of the most valuable raw data. At initial estimate it seemed like less than 1PB was enough for sure if focusing on text and not images/video. It even looked like maybe 200-400 TB might have been more than enough. I need to investigate more. I need numbers like 900GB for pile, then 2.5TB for a meta curated text only common crawl. So I’m assuming there is at least 10 of these in same ballpark that are important to have a copy of.
Any thoughts on the size requirements? Focusing on text.
https://www.home-assistant.io/blog/2022/12/20/year-of-voice/
That's not quite accurate, Microsoft first invested $1 billion in OpenAI in 2019 (four years ago) and another $10 billion a few months ago when it was valued at $29B.
So it's latest $10B investment gives it a 35% ownership, but it's unclear how much the $1B 2019 investment gave Microsoft. Some news reports peg Microsoft's current ownership at 49% of OpenAI, which means Microsoft's 2019 investment gave it 14% of the company (which would further indicate that OpenAI was valued at $7-8 billion at that time).
Of course like everything these days it's a cloud silo: the transcripts, videos etc are in their cloud. There's no way to have them simply posted to slack channels so they can be found later in searches etc.
The economic models and thinking of 2023 are seemingly still stuck in the PC era rather than being focused on the needs of the customer.
By the late 90s there were some interchangable formats for photos and audio at least, though often you still had to install plug ins, drivers, codecs etc for various proprietary formats especially video.
Basically a silo situation as bad as today.