Mistral: Our first AI endpoints are available in early access

Maybe in 1970 someone was arguing that Volvo is handicapped because Swedish safety regulations have become so stringent that their cars can’t compete on price elsewhere. Turns out it was exactly Volvo’s strength to be ahead of the curve on regulation.

Also, I'm not sure how much the EU is really going to matter for the future of AI.

You don't really need to conform to their regressive tendencies if you don't want to do business with the EU. Or you could dumb your stuff down for them if it ever comes down to it.

Generally, it just wouldn't be worth it to care too much about what the EU thinks.

My argument is opening up the commercial offering is perhaps the biggest story yet simply because of all excitement this company has been able to generate.

As for AI they have a lot of world-class talent but they somehow cannot transform that into a coherent product even though they did amazing stuff on the research side. It has been brought up many times on HN on how Google has a product-design problem and it seems to be true.

Google can't do anything about it either - consumers will prefer the chat approach and will go where it is offered. Google has to offer it, or people will go to ChatGpt and Bing, but at the same time they lose a lot of money each time people opt for the chat summary.

It is way expensive for Bing too, but they don't care because their market is growing and since the only thing people used to search Bing for was how to download Chrome and how to change the default browser, they aren't losing a lot this way.

Perhaps even sooner, given the pace tech is evolving nowadays comparing to when Google took over Yahoo's place.

I also know that because it’s open source, if I ever have a need to, I can host it on my own servers. Currently I don’t have that need, but it’s nice to know that it’s in the cards.

If you carefully craft and evaluate your more complex prompts against a closed model... and then that model is retired, you need to redo that process.

A lot of people were burned when OpenAI withdrew Codex, for example. I think that was a poor decision by OpenAI as it illustrated this exact risk.

If the hosted model you are using is open, you have options for continuing to use it should the host decide to stop offering it.

So, 10kwh could be a lot less than what you cite. That's also how grid operators make money. They generate cheaply and sell with a nice margin. Prices are determined by the most expensive energy sources on the grid in some markets (coal, nuclear, etc.). So, that pricing doesn't reflect actual cost for renewables, which is typically a lot lower than that. Anyone consuming large amounts of energy will be looking to cut their cost. For data centers that typically means investing in energy generation, storage, and efficient hardware and cooling.

If you consider 600w for the entire system, that's only 6kWh/1M token, for me 6kWh @0.2USD/kWh is 1.2USD/1M tokens.

And that's without the power efficiency improvements that an H100 has over the 4090. So I think 2$/1M should be achievable once you combine the efficiencies of H100s+batching, etc. Since LLM's generally dwarf the network delay anyway, you could host in places like washington for dirt cheap prices (their residential prices are almost half of what I used for calculations)

Here's how it works in reality:

https://docs.mystic.ai/docs/mistral-ai-7b-vllm-fast-inferenc...

FWIW, I most enjoyed the 29TB machine demo at the end.

I tried using it for "business document" use cases but have ran into this with code as well; the latter might be a better explanation given where we're having this discussion. If you only need the llm to retain the general shape of your inputs so it can reuse them to influence the output, the sliding context is fine. But if you need it to actually reuse code verbatim from the input that you fed it (or to remember the api calls and their surrounding context verbatim to recall from a sample of just one that this api must be called before that api, when the prompt includes instructions to that effect) the "decomposition" of the input tokens with the sliding model is insufficient and the llm completely fails at the assigned task.

It refers to the original Mistral 7B though not the new Mixtral fwiw

Again, sufficient training can overcome these limitations. But that's only for cases where the corpus of input documents is static or at least contains significant reuse.

Then they'd be differing opinions?