I wouldn’t trust anything geohot says. He doesn’t have access to any inside information.
It looks like at least one other person has also heard the same information.
That would negate, at least partially, the "we have 20 datacenters" advantage.
I don’t think they have an ensemble of 8 models. First, this is not elegant. Second, I don’t see how this could be compatible with the streaming output.
I’d guess that GPT4 is around 200B parameters, and it’s trained on a dataset made with love, that goes from Lorem Ipsum to a doctorate degree. Love is all you need ;)
Plus, none of the smaller models are really appreciably close to GPT-4 on most metrics. It's not clear to me you could get there at all with 1.3b models. Maybe somebody gets there someday with 65b models, but then you're far out of the reach of phones.
Also Apple has never gone for high RAM in mobile devices. I could go get 12GB in a brand new phone for $400, and some high end phones have had 16GB since 2020.
So combined, you could do normal app stuff with 2.5GB and 20-25b 3-bit inference with the other 9.5GB.
What is weird is how competitive the open-source models are to the closed ones. For instance, PaLM-2 Bison is below multiple 12B models[0], despite it being plausibly much bigger[1]. The gap with GPT-4 is not that big; the best open-source model on the leaderboard beats it 30% of the time.
That said, the advancements in models like Orca and the "Textbooks are all you need" paper are noteworthy (https://arxiv.org/pdf/2306.02707.pdf, https://arxiv.org/abs/2306.11644). I'm optimistic about what future smaller models could achieve.
I am writing a new book Safe For Humans AI in which I am constraining myself to using open models that can be run on a high end PC or a leased GPU server. Yesterday I was exploring what I could do with T5-flan-XXL, and it is useful, but not as effective tool as the closed OpenAI models. I will do the same with Orca this week.
Especially for LangChain, I recommend using ≥33B models like Guanaco or WizardLM. Guanaco-65B honestly feels on-par with ChatGPT-3.5. (To be clear, there is a large gap with GPT-4 though.) It is a costly test, although GPTQ (for instance on exllama) help make it affordable.
I haven’t tried Orca since they haven’t released the weights yet, but it doesn’t seem like they have a 33B version.
Oh it is though. I've tried several OS models and nothing comes even close to GPT-4. Turns out ClosedAI has a moat after all.
Wish there was a bookmark/remind-me function so I could come back in 8 months' time and see how close I was.
So tell me: how do you know the Geohot doesn’t have inside information?
There's no point in outing your sources, that's how you lose them.
Bluedevilzn made a negative claim, so naturally I'm asking if they can prove their negative.
To me he joins the ranks of the most basic shady crypto types.
Eh, is it? Not sure if I consider him an authority on anything anymore.
https://www.reddit.com/r/ProgrammerHumor/comments/z2y8i0/fro...
>This is the interview. Build this feature. You don't get source access. Link the GitHub and license it MIT.
is akin to "Build this for free, license it MIT so I can use it without any issues, and oh, btw, I dont have authority to hire you, teehee."
The right approach when you're new is to quietly pick a simple problem away from the core services where you can learn the processes and polices needed to get something into Production. More so when you're in a company that is undergoing a brain drain.
Then you can progressively move to solving more demanding and critical issues.
Before you run you learn to walk.
"Rocking the boat" is more likely than not to make things worse.
He was just clicking around compulsively, jumping between stuff randomly without even reading it, while not understanding what he was looking at. I have no idea how he's successful in tech, judging from what I saw there.
I think judging only from what you saw there is the issue. If you look somewhere like Wikipedia [0], you'll see he was the first person to jailbreak the iPhone, the first person to achieve hypervisor access to the PS3's CPU, he took first place in several notable CTFs (at least one time as a one-person team), he worked on Google's Project Zero team (and created a debugger called QIRA while he was there), creating comma.ai, and the list goes on.
Spent two weeks trying to find someone to build a faceted search UI and then quit.
a) It is valued at about a 1/4 of what it was purchased at.
b) Twitter Blue has generated an irrelevant amount of revenue and churn is increasing [1].
c) Roadmap looks poor. Video is a terrible direction where only Google, Amazon, TikTok etc have been able to make the numbers work and that's because it is subsidised through other revenue sources. Payments is DOA given Twitter's inability to comply with existing regulations let alone how difficult KYC/AML is to manage.
d) Regulatory and legal risk increases by the day. Lawsuits continue to pile up and EU/FTC are hovering around as Twitter is not in compliance with previous agreements.
e) Brand safety continues to be a huge challenge that isn't solvable without effectively going back to what Twitter was previously.
f) BlueSky and Instagram are both releasing competitor apps to the broader public in the coming months. The market simply won't sustain this many text-based social media apps.
b) Every social media company has a well populated graveyard of failed experiments behind them.
c) Opinion.
d) Business as usual for every social media company since the dawn of time.
e) Opinion. And advertiser behaviour suggests otherwise.
f) History is littered with new entrants which fail to unseat the incumbent. It does happen, but it’s statistically rare.
As well as comments from Musk himself about the decline in the value of Twitter and from advertisers themselves about the challenges around brand safety.
At best it has sped up towards that destination, at worst changes will eventually avoid it.
h) the tweets served are no longer weighted towards followers but instead whoever paid for blue checks
i) accounting on views is entirely wrong with a tweet registering a billion views
I don't disagree with him, it's just clear that he didn't understand how dire the financial situation was and that even a progressive refactor starting with the most basic features would take considerable engineering hours and money.
This has implications for what may be possible on consumer hardware.