You also need a shitton of data. People are saying that synthetic data is widely used, but pretty sure that is on top of the organic data. Both for pretraining and posttraining.
That said, I do think we will see more open and collaborative approaches over time.
Prior to AI making a PR involved considerable effort from a human. So the default position for many open source projects was that it deserved some level of attention for the effort. Even if many projects in practice would struggle to review every PR. But with AI tools this dynamic has shifted dramatically - many PRs have basically zero effort been put into it. Additionally there are many more of them, and often way bigger also.
Many people in LocalLLaMA Reddit community has been reporting the same, that 3.5 122B-A10B is on par or slightly better. And a 3.6 or 3.7 od the 122B is one of the models people want to see the most.
Dual 5060ti 16gb does over 100 tok/s on 35B A3B. Even with PCIE Gen 4 x4, which quite a lot of motherboards can do. Though Gen 4 x8 or Gen 5 x4 is slightly faster. Misc working notes on this hardware combo here, https://github.com/jonnor/embeddedml/tree/master/handson/mic...
Dual color filaments exist, and they do not mix at all... It gives the objects a nice transition when rotated. But indicates that color mixing in the nozzle is probably pretty difficult?
That OpenAI was in the wrong when they ignored everyone copyright, does not make it right to ignore their ToU. If a one wants IP and rule of law (incl contracts) to be respected, one should not violate others rights when it is convenient.
On a more risk-strategy level there is the size of their legal team, general endowment, and supplier and political connections to consider.
Everyone is free to ignore their ToU, but I can understand why a company would avoid it...
A 10'000 hour entry fee does rule out a fair bunch of people though, in practice. While there are few artificial barriers to learning to code, there still are some natural ones, like time.
I do not know which is easier. I am not sure that is even well established in research for generative text tasks whether a translation-first or native-language-first is the most sample efficient?
But for a national lab I think it is money well spent to figure out the possibilities and limitations of a native-language LLMs for languages with order of 5M-10M speakers.
These models will never compete with frontier models and do not need to - it is about hitting a good-enough, not being the best.
Behind the frontier, getting to a certain performance level, is getting easier over time - both sample and compute efficiency is going up.
Furthermore one can reuse investments in data (both agreements, infrastructure and datasets), compute (GPUs, servers) and know-how (training scripts, experienced engineers).
It would require an investment, but those will pay dividends later, as it becomes easier to train LLMs on/for Norwegian. If we need to translate everything to English we might as well just drop using Norwegian altogether. Practically everyone speaks English fluently already...
As a precondition I think we have to assume that the person in question 1) wants to learn and 2) is smart enough to absorb new info and apply it and 3) reflects enough to adjust their approach when hitting bottlenecks or making mistakes 4) has a drive to create. Without these, self driven learning is not viable - and that has very little to do with AI.
For such a person, I believe AI can be very empowering for learning. Like Google, wikipedia and stack overflow, Arxiv before it - AI tools give access to a lot of information. It allows to quickly dig deep into any topic you can imagine. And yes, the quality is variable - so one needs to find ways to filter and synthesize from imperfect info. But that was also the case before.
Furthermore AI tools can be used to find holes in arguments or a paper. And by coding one can use it to test out things in practice. These are also powerful (albeit imperfect) learning tools. But they will not apply themselves.
You are correct that bandwidth requirements depends a lot on the exact workload. And that in specific cases, it might be doable to have AM5 for multiple RTX6000Pro. The parent mentioned workloads that are general, and broader than inference-only. In that case I would consider spending a bit extra on the motherboard to ensure that PCIE bandwidth is not an issue.
There are likely _many_ paths to sustainable business models based on AI tech, that will come to fruition over the next decades. However whether they might not be as profitable as OpenAI and Anthropic are gambling on, is more uncertain.
Communication tech/tools enable more people to collaborate. It increases ability for labor that is far away from high value markets to contribute. Same goes for shipping tech wrt physical goods. On the global scale that is empowering the labor class.
Any productivity tool that individual laborers can purchase also (and that still needs the worker) is probably good for labor, overall.
This seems like a viable eval strategy. Presumably finding a bug requires some degree of understanding of the code, beyond just information retrieval.
However it probably does not measure things like prompt adherence or ability to create code that implements a specification?
Dynamic routing is the usual name for the piece that orchestrates which LLM will be used, based on query complexity. There in an open source implementation as part of the vLLM project (and probably others), it is a field of active research in several universities and labs.
It is also suspected that the frontier LLM providers might already be doing something like it behind the covers.
Hardware sales would be an excellent business model for open weights. Nvidia is already on it with their Nemotron models. Any new LLM/NPU hardware companies would want to so the same, if noone else does it for them (Chinese labs currently do).
Selling managed self-hosting solutions would be another. That is the business of that recent American company.
Selling fine-tuning services or similar adaptations is another. That is what Unsloth is going for, I believe.
Most likely any sound business strategy is going to be of "commoditize your compliments" type. There are many complementary products to open-weight - some probably not invented/discovered yet.
That is not at all the intention of the ARC team. By ARC teams definition, passing any single ARC-AGI benchmark does not mean that AGI has been achieved. Instead, AGI would be considered achieved when we are no longer able to come up with new benchmarks that the AI systems do not immediately do well on.
Partitioning is not all that expensive. It is definitely worth testing for your specific workload. We use TimescaleDB, which relies heavily on postgres partitions, have a bit under 100 million rows in our active set (last 90 days), across 120 partitions (device*time), and it works nicely. Over 100 partitions is probably a bit many for this workload, but since it works OK we have not changed it.
Network effect means it will be a huge and risky undertaking, and one needs to solve the bootstrap problem. But the costs of video delivery means that one would have to burn serious cash in the meantime. So it works in tandem.
TikTok kinda did manage to make a dent though - I suspect it substitutes for YouTube in some cases (though not all).
YouTubes biggest moat the last 10 years is probably more that all the viewers and creators are already there. Any competitor has a huge disadvantage - creators are not interested in a place without viewers, and viewers not in a place without creators/content.