I think it's a direct translation of the French phrase "mixité sociale" mentioned in the article, I bet the connotations are slightly different in English than in French.
1,069 karma · joined July 29, 2012
> 1979 Feb – May one way – $154 – $169 /Sept – Oct one way – $194
> 1980 Feb – one way – $227
so more than $900 today to fly one way on a flight that today you can have for $40 one way on a budget carrier! I guess I don't mind the extra hour it takes on a 737 or A321.
Although the more fair comparison would be to the competing prices at the time, not today's prices - any ticket was quite a bit more expensive in 1979-80, so that factors in.
* There is a class of problems which are harder than polynomial time complexity to solve, but are not np-complete
* LLMs will generate an "answer" in formal language to this class of problems posed to it
* LLMs can at most solve problems with polynomial time complexity due to their fundamental design and principles
* Therefore, LLMs cannot solve > polynomial problems and not np-complete problems either
All of which I buy completely. But I think what people are more interested in is, why is it that the LLM gives an answer when we can prove that it cannot answer this problem correctly? And perhaps that is more related to the commonsense notion of hallucination than I first gave it credit for. Maybe the reason that an LLM gives a formal language answer is the same reason it gives a hallucinatory answer in natural language. But I don't think the paper sheds light on that question
I think this is a clever point and an interesting philosophical question (about math, computer science, and language), but I think people are mostly trying to apply this using our commonsense notions of "LLM hallucination" rather than the formal notion they use in this paper, and I don't see an obvious connection, since commonsense hallucinations (eg inventing chapters of a novel when asked to produce summaries, inventing specific details when asked) don't seem to be NP-complete problems but rather are hallucinatory for some other interesting reason. (I apologize if I have not captured the paper correctly and would welcome correction on that, I read it quickly)
The statement that the formal world (the world of math and logic and formal grammars) is a subset of the "real" world (or perhaps, the world of natural language) is really interesting to me as well. Most humans can't solve formal logic problems and parse formal grammars but don't suffer from a (strong) hallucination effect, and can work in natural language in great proficiency. Is hallucination inevitable in humans since we also can't solve certain NP-complete problems? We have finite lifespans, after all, so even with the capabilities we might never complete a certain problem.
Why? I think this is because books3 contains, you know, books - including some really long books, and proof-pile contains math papers and math stuff, which isn't as long.
So overall I think what you're seeing is a general trend of increasing perplexity on windows above 256k, between 256k-2048k, which is probably not so surprising - or at least, not so surprising when you consider the context of the paper, which is taking a model pre-trained with a much shorter context window and extending the context window using a novel technique. It's hard to adapt a model trained to do one thing into doing another thing, and that's what they're doing, so in that context, it tracks that the longer the context window, the worse the performance.
Good: Put a meeting on a calendar in the future in order to force all discussion and investigation to happen before that meeting and keep things moving.
Bad: Adding a "decision meeting" doesn't make a decision any easier or harder than before, and doesn't make the research and investigation take any less time than before, and often we don't know what we don't know, so beginning research leads to more research...
I think it is really useful for making decisions that people just don't care that much about and aren't that invested in, since it motivates a timeline for those decisions. Otherwise, it just creates an arbitrary deadline. Why are you working late on a Friday night? Oh, well, my manager created this deadline on Monday morning to make this decision.
Actually my point is that to specifically isolate costs that are from AI is a lot of work at a company of this scale. These companies are absolutely accountable for their overall energy usage and externalities!
> Moreover, the organizations best placed to produce a bill — companies like Meta, Microsoft, and OpenAI — simply aren’t sharing the relevant information.
To me this shows both unfamiliarity with large corporate structures and unfamiliarity with AI research. The former because, if you want an exact accounting at a company where there are several teams running dozens or hundreds of models, it becomes someone's job essentially to compile this information because it takes a lot of work. So, you are surprised that a corporation doesn't outlay $200k/year or more getting these figures to decorate your article with? The corporations do know, on a month-to-month basis, what they're spending on this stuff - since they settle the invoices. But doing the work to get these figures into a simple, digestible form is a lot of effort, and I think that quite frankly, that effort should fall on the journalist, since it's the journalist who gets the benefit from having those numbers...
As to unfamiliarity with AI research, many papers I have been reading lately are very interested in measuring and minimizing the compute cost of models, and they often compare different methods and are often extremely precise about their training process and equipment. I feel like this article wants to sell the story that these faceless corpos don't care about energy consumption, but the researchers definitely do. Granted certain specific products / models such as ChatGPT 4 do not disclose exactly their process, but I feel like it would not be difficult to come up with a good estimate using similar models (mostly documented right out in the open in scientific papers).
Tracking the energy consumption of AI is an important and emerging issue, but this article feels too partisan to be useful.
I think what you will discover is that network bandwidth is a finite resource you have to pay for and ration out to your customers, and that the easiest way to finitize and ration something your customers expect (errantly) to be infinite is to charge for it, and charge exactly that price that will cap demand!
The existence of the world wide web has tricked us into thinking that sending traffic anywhere is/should be the same, but of course it is not. So while the price you (a cloud customer) pay for egress pricing is (often) indiscriminate on where that traffic is going, using common sense, we can understand that some traffic is more expensive than others, and the price we pay is a blended price with that aspect "baked in" or "priced in".
If you want an un-influenced "search for knowledge", you are well within your rights to pursue that independently using your own model and software, but because you are accessing software developed by another company, these rights do not apply.
There's a joke online - "nobody speaks more confidently than a 15 year old who has smoked weed twice explaining how to smoke weed to a 15 year old who hasn't smoked weed before" - nobody writes an angrier blog post than someone who has watched two YouTube videos about a subject.
Doing the work to improve society is boring and slow. It's talking to people in the community. It's local politics. It's generational shifts. It's a warm smile to the older person in the store, in the hospital, as we try to move the needle ever so slightly in the right direction.
That's not to say we shouldn't consider the consequences of these changes and provide incentives and disincentives to nudge things one way or another, and manage the demographic shifts in order to lessen their shocks.
The author seems to think so, and indeed, with today's GPTs, that is often what you might get back. But I think there's a lot more going on here - with natural language, a lot of the training effort goes into producing natural language output which conforms to the formal rules of language (it is intelligible, syntactically correct or roughly so, follows grammar, and semantically "about" what it is supposed to be about). But the interesting thing about natural language is that there are infinitely many correct responses which are valid formal responses to the prompt, both ones that introduce new or rare creative possibilities and ones that tread the same ground as a wikipedia article, for example (which is usually intended to be very neutral). So an LLM can produce both types of responses - formally correct and boring, formally correct and interesting - and you can see this today by offering "mix-ins" to the prompt like, "Tell me about Saudi Arabia's ruling family, in the style of Peter Griffin". Of course, it won't select to use the style of Peter Griffin without that prompt, but I think you can imagine a "weirdo" LLM that is trained with some positive response toward creative or interesting output too.