6,603 karma · joined March 13, 2013
How might performance compare between:
1.) Using a model like Gemini to load it all into context at once
2.) Using one of the various summarization systems/embedding/RAG etc.
3.) Fine tuning the whole code base into Gpt-4o
If it can be overcome in existing models, it’s probably going to involve different aspects including vocabulary, style, and organization.
Have the last MacBook Pro Core i9 model with 64GB RAM and 4TB SSD and in trying to sell it the prices seem super low way less than 1k.
I bet the M2 is worth way more.
Wonder if there’s any way to intuit this before working it out.
But there’s been a huge amount of questionable behavior and there has to be personal responsibility with that. It’s not an overstatement to call this part of one of the biggest failures of science in history, and you can’t just sweep that under the rug as unintentional.
As far as pop science I’m not addressing that but those sins don’t exonerate everything else.
I don’t get what you mean about Popper either, he likely would’ve been all over the reproduceability crisis and calling out integrity as a key issue.
Yes, science is self-correcting and things have definitely started to improve after learning from all of this. But the damage has been done. at the time when we need science, the most it’s been discarded by a significant part of the population.
Even worse, the replication crisis is only one reason that the public has continued to lose faith in science in the post truth era.
It’s also the disinformation campaigns that set out to attack whatever’s in a groups interest whether it be politics or the environment.
Maybe the coup de grâce will be social media which encapsulates people into bubbles seemingly impenetrable to the truth.
What you're saying sounds plausible, but I don't see how we can conclude that definitively without at least some empirical tests, say a set words that predictably give an error along token boundaries.
The thing is, there are many ways a model can get around to answering the same question, it doesn't just depend on the architecture but also on how the training data is structured.
For example, if it turned out tokenization was the cause of this glitch, conceivably it could be fixed by adding enough documents with data relating to letter counts, providing another path to get the right output.
Tokens aren’t the source of facts within a model. it’s an implementation detail and doesn’t inherently constrain how things could be counted.
With some better/clever prompting I’m not sure they could be as easily detected now.
So many smart, credible people build up lines of thinking (a necessary step and best that can be done presently) around subjects. These get discussed, reported on, worked with, and very easily become conventional wisdom. Yet so often they turn out to be way off course.
This topic has fired every one up because it’s unnecessarily cruel, hurts families who didn’t do anything wrong, enriches companies not providing any value, and shows people trying to be “tough on crime” when very ironically they’re probably creating more crime by eroding support systems.
The parent commenter mostly expresses that outrage, and makes a passing comment about business competition.
By this time anything you said that could be perceived as possibly being near the other side of the argument is going to be taken as supporting the other side.
But they are two separate points that can be independently discussed you say? Technically that’s true, but humans don’t work like that.
Always step back and look at the biggest point being made and realize, there may be little room for nuance depending on the context.
Yet there’s people on YouTube self administering gene therapy to cure lactose intolerance.
Not an endorsement, just to say it’s interesting how easy it’s becoming to do these things.
I think I’ve had an unusual number of people close to me choose that option, including my father recently so I guess I’m still trying to reconcile it all. Their pain vs mine, what could have played out differently, etc.
I sincerely wish the best outcome for you and yours.
But then after Oppenheimer did the math in 1930 and showed that collapse into a black hole was within the theory of relativity would you still taken the same position?
Is this a contradiction or am I misunderstanding something?
Btw overall very impressive work great job.
Paragraph 7 acknowledges what the parent comment mentioned, that other species use very little oxygen.
The title says this one needs none. The 2nd paragraph says:
“…it doesn't breathe; in fact, it lives its life completely free of oxygen dependency.”
If that’s not enough, it’s also made clear in Wikipedia:
Yet apparently to this day he draws over 200k/yr in salary from Berkeley. I believe they are not entirely funded by tuition/endowments which means California tax payers support him at least in part.
To add to the collective anecdata, here Gpt 4.o does fine and Claude invents packages that don’t exist:
Question:
“Write code in Swift to use RAG and LLMs to allow users to ask questions about a document collection.
Let’s use services to get the app completed quickly.
What do you think about using Cohere for the text embedding model, and Pinecone for the vector db?”
Output (packages don’t exist):
import Cohere
import PineconeSwift
class RAGService {
private let cohere: CohereAPI
private let pinecone: Pinecone
- The overall retraction rate is 4 in 10,000.
- Most researchers go their entire career without a retraction
- She now has 4.
What’s the probability of a major collision? What’s the probability that ISS significantly damaged?
It says there’s no consensus from the models, so we can’t know what any of them predict?
1) It’s weird to choose linear attention for their implementation because that’s not what their paper is about and they claim no insights relevant to attention mechanisms.
2) By benchmarking all models this way (linear vs linear) it likely inflated their numbers over comparing their removal of matmul in a quadratic vs quadratic scenario.
3) This claim implies a comparison to the state of the art in language models where the standard is quadratic attention, and is therefore a flawed comparison:
“We processed billion-parameter scale models at 13W beyond human readable throughput, moving LLMs closer to brain-like efficiency.”
4) Those type of brain comparisons fall apart under scrutiny, are not standard in ML research and don’t mean much anyway.
5) Right up front in the abstract they make specific performance claims and imply they come from removing matmul, but don’t mention linear attention until section 4 on experiments.
Automatic proof generation is a massive open problem in all of computer science and not close to be solved. It’s true LLMs aren’t great at it and more is required for example as with the geometry system Deepmind progresses on.
On the other hand they can be very useful to explain concepts and allow interactive questioning to drill down and help build understanding of complex mathematical concepts, all during a morning commute via the voice interface.