espadrine

Past Mistral investors: JC Decaux (urban advertizing), CMA CGM CEO (maritime logistics), Iliad CEO (Internet service provider), Salesforce (client relation management), Samsung (electronics), Cisco (network hardware), NVIDIA (chips designer)[0]. I agree ASML is a surprising choice, but I guess investments are not necessarily directly connected to the company purpose.

BTW, I generated that list by asking my default search engine, which is Mistral Le Chat: indeed, using Cerebras chips, the responses are so fast that it became competitive with asking Google Search. A lot of comments claim it is worse, but in my experience it is the fastest, and for all but very advanced mathematical questions, it has similar quality to its best competitors. Even LMArena’s Elo indicates it wins 46% of the time against ChatGPT.

[0]: https://mistral.ai/fr/news/mistral-ai-raises-1-7-b-to-accele...

At least it is not unprecedented. Palantir raised a series I in 2020 after 17 years of operation.

It would be interesting to have two generations per model without cherry picking, so that the Elo estimation can include an easy-to-compute standard deviation estimation.

The best model there is 2.5B parameters. I can believe that a model 10x bigger is somewhat better.

One element of comparison is OpenAI Whisper v3, which achieves 7.44 WER on the ASR leaderboard, and shows up as ~8.3 WER on FLEURS in the Voxtral announcement[0]. If FLEURS has +1 WER on average compared to ASR, it would imply that Voxtral does have a lead on ASR.

[0]: https://mistral.ai/news/voxtral

I agree that there are some robotic designs that unnecessarily mimic human limbs. I have in mind heads, and feet (instead of wheels).

A hand however, is useful because so many manufactured objects have been constructed for their purpose.

Here is someone that had significant corruption until they stopped: https://www.xda-developers.com/why-not-to-spin-down-nas-hard...

There are many similar articles.

Easy yes. Even VPS providers need to maintain the IP, since your DNS typically points to that IP. You can also typically move the IP to another machine from the same provider.

But as a resut, VPS often have a different price for public IPs compared to private IPs. For instance, it costs €0.004/h per IP at Scaleway.

> I rather define freedom by the government not deciding what's good for me

Does that mean you are against this bill?

Before the bill, your community could either use your own water system, without fluoride, or use the wider system, which has fluoride.

After this bill, your community no longer can use your own water system with fluoride, and the wider system also does not have fluoride.

On a first read, this bill makes the government remove a choice for you, deciding what is good for you.

(I don’t have a horse in this race honestly. Assuming everyone can get toothpaste and toothbrushes, the effect is the same. But the wording of the bill is strange: “may not add fluoride” rather than just “is allowed not to add”.)

Source: https://archive.is/Nustz

> I still haven't seen any good reasoning for why NASA would delay the return flight

So many comments spread unsourced assertions on the topic on both sides. Let me change that.

On 24 Aug 2024, NASA stated in a conference published on X[0]:

> NASA has decided that Butch and Suni will return with Crew-9 next February.

So the decision sounded like it stemmed from NASA, and the plan a year ago was for a return in the time frame that actually occurred.

On 28 Sept 2024, the spacecraft that would bring Ms Williams and Mr Wilmore was launched. They restated the same plan[1]:

> A SpaceX Falcon 9 rocket and a Dragon spacecraft will launch Crew-9 to the space station for about a five-month mission. Hague and Gorbunov will join Butch Wilmore and Suni Williams, who are already aboard the space station, and all will return to Earth as a crew of four in February 2025.

Thus there was no action after Mr Trump took office that changed the plan. (Note that the SpaceX article we’re commenting on has a mistake, stating that Mr Grebenkin has come down on 18 March, but Mr Grebenkin came down on 25 Oct; it is indeed Gorbunov that came down.)

On 7 March 2025, Ken Bowersox, associate administrator, Space Operations Mission Directorate, stated[2] on the motivation for this:

> When it comes to adding on missions or bringing a capsule home early: those were always options but we ruled them out pretty quickly just based on how much money we've got in our budget and the importance of keeping crews on the International Space Station.

When asked specifically about a request from Mr Musk to have an earlier return:

> Was anyone outside NASA or the White House involved in the decision not to bring Suni and Butch back sooner? — There may have been some conversations that I wasn't part of. When we made the technical decisions about Starliner […] our leadership at NASA was trying to make sure that we considered everything just at a technical level and that's what we did.

[0]: https://x.com/NASA/status/1827396382702878966

[1]: https://www.nasa.gov/blogs/commercialcrew/2024/09/28/nasas-s...

[2]: https://www.youtube.com/live/Xq5CH-d4IuU?si=-q-J65i7G_d6Mcxs...

In RAM, yes. But if you compute an activation, you need to load the weights from RAM to the GPU core.

There is no question that quantization degrades quality. The GGUF R1 uses Q4_K_M, which, on Llama-3-8B, increases the perplexity by 0.18[0]. Many plots show increasing degradation as you quantize more[1].

That said, it is possible to train a model in a quantization-aware way[2][3], which improves the quality a bit, although not higher than the raw model.

Also, a loss in quality may not be perceptible in a specific use-case. Famously LMArena.ai tested Llama 3.1 405B with bf16 and fp8, and the latter was only 2 Elo points below, well within measurement error.

[0]: https://github.com/ggml-org/llama.cpp/blob/master/examples/q...

[1]: https://github.com/ggml-org/llama.cpp/discussions/5063#discu...

[2]: https://pytorch.org/blog/quantization-aware-training/

[3]: https://mistral.ai/news/ministraux

> if a llm will run with usable performance at that scale?

Yes.

The reason: MoE. They are able to run at a good speed because they don't load all of the weights into the GPU cores.

For instance, DeepSeek R1 uses 404 GB in Q4 quantization[0], containing 256 experts of which 8 are routed to[1] (very roughly 13 GB per forward pass). With a memory bandwidth of 800 GB/s[3], the Mac Studio will be able to output 800/13 = 62 tokens per second.

[0]: https://ollama.com/library/deepseek-r1

[1]: https://arxiv.org/pdf/2412.19437

[2]: https://www.apple.com/newsroom/2025/03/apple-unveils-new-mac...

The blind test at lmarena.ai does give it a higher Elo than GPT-4o (API), Claude, and Gemini 1.5 Pro. It seems that people do enter real-life scenarios in the arena.

Hernan Moraldo is from Argentina. That may be all there is to it.

> I'm quite nervous for the future.

Videos like these were already achievable through VFX.

The only difference here is a reduction in costs. That does mean that more people will produce misinformation, but the problem is one that we have had time to tackle, and which gave rise to Snopes and many others.

Philosophically, it always bugged me that distros were so centrally in control of packaging: app developers are not allowed to package them in the way they see fit, which has caused some friction in the past[0]. It seems healthier and more scalable if each developer could package it autonomously without work from the central maintainer group.

KDE Project Banana mentions little about packaging, apart from this:[1]

> Instead of legacy packages we target modern deployment systems such as flatpak and systemd-sysext

I hope they succeed in that respect. Flatpak has the promise of decentralizing distribution. That said, my experience of it so far has been brittle, with things like changing your locale or shell prompt causing your apps to freeze[2]. I hope they make a stable operating system nonetheless, but it is not easy.

[0]: https://news.ycombinator.com/item?id=41407768#41408628

[1]: https://community.kde.org/KDE_Linux

[2]: https://github.com/flatpak/flatpak/issues/5564

When Meta prevented the EU from using meta.ai or even downloading its vision models, I sunk my head in the AI legistation.

Here, I am honestly not sure which part they rely on, to say that what they made might be unlawful.

The closest thing I found for Meta was that “emotion recognition systems” are classified as high-risk (paragraph 54), and high-risk systems must have their training data disclosed (Art 11(1))[0]. In theory, you could upload photos to meta.ai and ask it what emotions are displayed, but it is already a stretch. For GenChess, I’m at a loss; it doesn’t sound like you can do that. (Not that it prevented any vision chatbot from releasing.)

If someone has a better guess as to why they might have restrained it here, I am curious.

[0]: https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=CELEX%3A...

> Furthermore, by rotating the vector, we have absolutely zero impact on the norm of the vector, which encodes the semantic information of our token.

Doesn’t the angle encode semantic information? Cosine similarity works for embeddings after all.

You’re right. I understood it to require taking the top 2^30 tokens, but instead they sample 2^30 times with replacement.

Too bad they only formulate the detection positive rate empirically. I am curious what the exact probability would be mathematically.

The academic paper: https://www.nature.com/articles/s41586-024-08025-4

They use the last N prefix tokens, hash them (with a keyed hash), and use the random value to sample the next token by doing an 8-wise tournament, by assigning random bits to each of the top 8 preferred tokens, making pairwise comparisons, and keeping the token with a larger bit. (Yes, it seems complicated, but apparently it increases the watermarking accuracy compared to a straightforward nucleus9 sampling.)

The negative of this approach is that you need to rerun the LLM, so you must keep all versions of all LLMs that you trained, forever.

That makes me wonder though what the best loss function was. I assume you used MSE on the logscore. I wonder if a sigmoid on which of two articles has the higher score would yield better results for the downstream RLHF task.

It takes no time at all to find other major mistakes. For instance, the Mixtral diagram § 6.6.1 shows a single router that selects separate 32-layer transformers. Instead, Mixtral has one router per layer (inside of each block), and it doesn’t select a transformer block: it selects a feedforward.

Terminology.

Since they targeted very low risk, they did a geographically-segmented rollout, starting with Phoenix, which is one of the easiest places to drive: a lot of photons for visibility, very little rain, wide roads.

At this week’s dotAI conference, Ines Montani (who works on the SpaCy project) highlighted this ex-job as a warning to AI builders, so that they do not work on systems that have no future, because better and cheaper alarm clocks (for knocker-uppers) are coming.

In particular, she saw chatbots as being an inefficient user interface that would eventually be replaced by better integration between assistants and conventional UI.

KataGo has a special model weights release with human-like play at various Elo: https://github.com/lightvector/KataGo/blob/master/docs/Analy...

You can see in the release notes a few screenshot examples where a particular move changes likelihood as you get to higher-level play: https://github.com/lightvector/KataGo/releases/tag/v1.15.0

> Hard to see how can Mistral compete with Meta

One significant edge: Meta does not dare even distribute their latest models (the 3.2 series) to EU citizens. Mistral does.

Five years to build each SMR is faster than what it takes to build a typical EPR (from 10 to 20 years), but it is still longer than what I expected. Besides, the 5-year figure might have overruns as they work through regulatory hoops towards a permit.

Nevertheless, before even having one built, it will allow Google to say that large model training will eventually not produce CO2.

This is a topic I love to study. The mathematical analysis is reasonable; the policy gradient is a classic approach; I love Sutton’s RL book on it: http://incompleteideas.net/book/RLbook2020.pdf Even though nowadays many people rather use cross-entropy for training RL, which I believe leads to more stable training.

Some of the equations feel a bit imprecise. I prefer to use random variables (rather than the U term), so that uncertainty permeates every value. As a result, the epsilon-t formula might be a perfectible fit (it goes negative past a certain t, which is unrealistic).

The Pixtral report[0] compares positively to Molmo.

(Also, beware, molmo.org is an AI-generated website to absorb through SEO Allen AI’s efforts; the real website is molmo.allenai.org. Note for instance that all tweets listed here are from fake accounts since suspended: https://molmo.org/#how-to-use)

[0]: https://arxiv.org/pdf/2410.07073

That is a concern that is shared with ReLU. But since the weights are shared across the context/minibatch, perhaps that would not be an issue, similar to ReLU.