WanderPanda

Yes, it seems like capability is logarithmic wrt compute but utility (in different applications) is exponential (or rather s-shaped) with capability again

It needs to be intact, the pdf is rendered by the arxiv backend based on the source

I mean it is humanity’s LAST exam. Humanity’s first exam would probably be something about communication? Or about building and predicting effects of certain tools?

It’s sad that we have to assume this is the agenda for all offerings that don’t allow paying current price x years in advance (possibly including adjustion for expected inflation). Leaves a very sour taste, especially in the case of hardware devices

Or is it just white collar workers experiencing what blue collar workers have been experiencing for decades?

Crazy conglomerate discount on Alphabet if you can see TPUs as the only Nvidia competitor for training. Breaking up Alphabet seems more profitable than ever

No you don‘t need much bandwidth between cards for inference

Damn good to know! Have been gaslit by the ever-changing docker install instructions. Of course it would be a lagging version, but I think the docker feature-set has converged years ago, why would I care any more about the docker version than e.g. about the version of grep?

Makes me wonder why docker still didn't make it to the ubuntu/debian repositories. Would be such an easy net benefit

This sounds like the (debunked) labor theory of value :p

I wonder why that is? because they are trained with dropout?

+ being early on capacitive touchscreens and multi-touch

You can just alt + click the audio options in the control center to select the input source

JAX still has the

"This is a research project, not an official Google product. Expect bugs and sharp edges. Please help by trying it out, reporting bugs, and letting us know what you think!"

disclaimer in its readme. This is quite scary, especially coming from Google which is known to abandon projects out of the blue

How will we even measure this? Benchmarks are gamed/trained on and there is no way that there is much signal in the chatbot arena for these types of queries?

I think in just a few month the average user will not be able to tell the difference in performance between the major models

I'd be shocked if we don't see diminishing returns in the inference compute scaling laws. We already didn't deserve how clean and predictive the pre-training scaling laws were, no way the universe grants us another boon of that magnitude

If the writing & arts vs. doing laundry & cleaning dishes is any indication, it does not look rosy. All the fun and rewarding parts (low hanging fruits / quick wins) of coding might be automated. What remains are probably things like debugging race conditions in legacy systems and system administration etc.

The St. Petersburg paradox is where hypers and doomers meet apparently. Pricing the future infinitely good and infinitely bad to come to the wildest conclusions

Doesn‘t even have to be about money. Would be perfectly rational to comply to minimize uncertainty, public outcry, getting into the cross-hair of other countries‘ regulators and distractions from operating the core product/service. In short: for most people it wouldn‘t be worth the headache to stand up against a state like this.

This is one of these technologies that is indistinguishable from magic.

Agreed, on another note I also struggle to see how no one could create a (better) CUDA implementation for e.g. $1B engineering budget

I washed my Airpods in a washing machine once and they continued working normally for 6 months until the noise cancelling started to fail on one side.

I found it fascinating how far we can go with even just < 10 kilobytes per second [1]

[1] https://bellard.org/tsac/

I would have expected fine-tuning to be good at imparting knowledge. Pre-training is often done for a single epoch only and models soak up the knowledge like crazy without multiple passes so why would fine-tuning be any different?

Compiler folks: Is there any chance compilers will be able to find optimizations like FlashAttention on their own? Seems like TVM and tinygrad are working in that direction but I find it hard to believe that that would be feasible

I enjoyed the docu recently and I‘m still in awe how „software-defined“ the voyager system was/is for it’s time

Also that level of hustle is not (yet) common in Germany I believe

Do you mean emergence and network effects IN machine learning? Otherwise I would be very curious how machine learning fits that list in your view

I‘m using Raivo. It hasn’t let me down, yet

Plus all the flexibility of CUDA is not needed for LLMs anyways