aubanel

> For me that's just a great example of how basically the vast majority of AI influencers (who vie for influence on social media, rather than research) are basically clueless about AI and CS

This is a bit stark: there are many great knowledgeable engineers and scientists who would not get your point about a^nb^n. It's impossible to know 100% of of such a wide area as "AI and CS".

Net carbone negative is bullshit (no such things as net emissions if you don't have a "removal" side, which is not happening rn)

Then nuke is much better for low co2 energy mix, cf France vs Germany

So your point is: 1. The LLM revolution had no significant positive impact so far 2. A dozen areas (phishing, web search, web content, images, education, science, social media) were completely changed by LLMs for the worst

Your point 2 seems to indicate that the potential for impact is huge. People also think of positive impacts, not only negative ones, that's why they invest.

FYI, some sections are LLM-generated bullshit: https://x.com/sam_paech/status/1848332471953448972

The example shown in the tweet has been edited out of the paper since, but there must be others. High noise, low signal content.

I disagree with this: when an actor is known for repeated bad faith, it's often a net gain of time to dismiss what they say by default. Only if someone else presents the same argument should you take the time to analyze it.

Yes but the article's setting is precisely about 1 email so 1 inference, and their number is 0.14kWh, which is way off.

In the collection of Rilke's letters "Briefe an einem jungen Dichter", he writes what I think one of the best advice about writing a book or not:

"This above all: Ask yourself, in your night’s quietest moment, Do I have to write? Dig deep down into yourself for the answer. And if it is yes, if you can meet this solemn question with a strong and simple “I must,” then build your life around that necessity."

It's harsh, but I think the necessity aspect is important: if you feel like a spring from which the urge of writing naturally flows and you could not repress it, then write. Else, don't.

He's speaking about his objective to make ever stronger LLMs: so for this his secondary objective is to measure their real performance.

The human preference is not that good of a proxy measurement: for instance, it can be gamed by making the model more assertive, causing the human error-spotting ability to decrease a lot [0].

So what he's really saying is that non-rigorous human vibe checks (like those LMSys Chatbot Arena is built on, although I love it) won't cut it anymore to evaluate models, because now models are past that point. Just like you can't evaluate how smart a smart person really is in a 2min casual conversation.

[0]: https://openreview.net/pdf?id=7W3GLNImfS

GPT-4o also has vision capabilities: https://platform.openai.com/docs/guides/vision

I see, thank you for the explanation! I agree your translation is better for removing the pompous aspect of the very original translation.

As a native French speaker (maybe you are too), I think GP understood the nuance you mentioned but still proposes a good translation to reflect less pressure.

Maybe it's an occurrence of Thucydides trap: the reigning power is afraid to lose its crown, and as a result becomes more and more aggressive towards the growing power.

It seems to me that many Americans are personally sensitive about these issues of power. It's also the case with Russia : I agree that Russia is doing terrible things, but people tend to see the Russian enemy everywhere. Cf the arson on train lines in France at the beginning of the Olympics: even here where people are normally rational, every single comment was focusing on Russia and ignoring other possibilities. But actually it was done by French far-left activists.

Impressive to see the positivity that both him and his wife Bess maintained in their last posts, despite the constant pain and oncoming death. Real strength.

Rest in Peace Jake, and I wish you the best on the path forward Bess!

Great, now I can finally find the top 5 hiding places for these friggin campers!

Just to be sure: the post does not say that the performance of Open LLMs is plateauing (because that would be false, e.g. Google just released a Gemma2 that blows out of the water all previous open models from the same sizes).

Its true title is "Performances are plateauing, let's make the leaderboard steep again", which means "on the Open LLM leaderboard, top models have basically reached a point where they've all grokked the benchmarks which makes it harder to distinguish them, so let's change the benchmarks for harder ones to make a difference again."

LMSys Chatbot Arena is a crowd-sourced ranking with an ELO system: basically users a presented with 2 hidden models, they get the answers of the 2 models when presenting their request, and they vote which one performed bests, which realized one marche and updates the ELO scores. This is the closest thing that we have to a gold truth for LLM evaluation: and Gemma2-27B performs extremely well in Chatbot Arena ELO.

It's exceptionally strong. In LMSys Chatbot Arena, the 27B version scores above LLama-3-70B, at the level of OpenAI GPT-4 and Claude-3 Sonnet!

Something great I've specifically witnessed yesterday: when doing a small request, like ask for a feature request in the team channel, don't just describe a small use case.

Instead, provide the bigger picture: e.g. "I'm currently trying to climb that benchmark, I'm close to the top and only need a few % points that I could get with this new feature"

For me it worked wonders: a PR that I had been awaiting for some time was done overnight by an enthusiastic and helpful coworker!

This model has: 1. On par or better performance than Llama-3-70B-Instruct 2. A much more comfortable context length of 128k (vs the tiny 8k that really hinders Llama-3)

These 2 feats together will probably make it the first serious OS rival to GPT-4!

Previous post about the version 1 of Agents: https://news.ycombinator.com/item?id=35889743

We've done a complete revamp of the package since then, with two goals in mind: - clarity through simplicity: as few abstractions as possible, accessible prompts - modularity: let you free to use any building block you want

We also added very performant ReAct agents, which allows to get a good rank on the GAIA Leaderboard, and even to beat GPT-4 based agents with a Llama-3-70B agent!

Agents 2.0 is compatible with LangChain tools out of the box, which should make it easy to insert in most agentic workflows.

Please elaborate on this: how is it flawed?

> Additionally, members of the program receive priority placement and “richer brand expression” in chat conversations

This sounds particularly bad since it's the polar opposite of what Sam Altman himself pretended to want in his recent Lex Fridman ITW (March 17):

> I like that people pay for ChatGPT and know that the answers they’re getting are not influenced by advertisers. I’m sure there’s an ad unit that makes sense for LLMs, and I’m sure there’s a way to participate in the transaction stream in an unbiased way that is okay to do, but it’s also easy to think about the dystopic visions of the future where you ask ChatGPT something and it says, “Oh, you should think about buying this product,” or, “You should think about going here for your vacation,” or whatever. > (01:21:08) And I don’t know, we have a very simple business model and I like it, and I know that I’m not the product. I know I’m paying and that’s how the business model works.

Wow that's disgusting.

We generally agree after some criminal decisions that decision-makers should be punished, and political parties should be forbidden.

Why not apply this to companies ?

I see the benefits (punishing stakeholders and deleting an institution that did terrible use of its power), and the downside (losing jobs) does not seem that terrible.

Putting "cut-throat bureaucrats" in the "do not care for impact" side of the axis seems unnatural to me: I think these people do care for impact, and that's why they are so decided about imposing their ways. But their definition of impact is "doing things the right way", which corrupts their want for improvement into a pile of processes.

On WizardLM2: the weights have been pulled back by Microsoft from HF Hub, as there seems to be internal debate on whether to really publish it. There's a risk that it does not ever get published again. So keep your local copies if you have any!

The calls with long context are done through specific APIs, that you can call for instance in Python or Javascript.

Here's a quick start guide with OpenAI: https://platform.openai.com/docs/quickstart?context=python

> I think in summary, a better approach would’ve been “You obviously know the 50 states, GPT, so just give me the full name of the state this pertains to, or Federal if this pertains to the US government.”

Why not really compare the two options, author? I would love to see the results!

I think agents are not a fad, they're here to stay since using an LLM in an agent system is the only way to let it access real world task, which they will end up doing when they're good and reliable enough.

That said, I believe the current best models are still not good enough - but let's wait a few months.

Even in the most restrictive définition of recall as in "retrieve a short contiguous piece of information inside an unrelated context", it's not that good. It's always the exact same needle inserted in the exact same context. Not the slightest variation apart from the location of the needle.

Then if you want to test for recall of sparse information or multi-hop information, it's useless.

Be aware the haystack test is not good at all (in its current form). It's a single piece of information inserted in the same text each time, a very poor measurement of how well the LLM can retrieve info.