gdiamos

Scaling laws assume the error metric and data distribution.

There is a lot of follow on work that explains what happens as you change them, e.g. Scaling Laws for Transfer - https://arxiv.org/pdf/2102.01293

I think it’s fortunate that transfer works in a similar way.

Common crawl (and Reddit, stack overflow, etc but not 4chan) was much easier to get access to at the time than using mechanical Turk.

There is certainly room for more work. There were many papers on scaling laws in NeurIPS this year.

When I first saw scaling laws in that deep speech experiment notebook, I didn’t believe it could be real. I was worried for months that we made a mistake, or that it only worked for that one dataset.

I started to believe it after we (Joel Hestness in particular) reproduced it in so many experiments in “scaling is predictable empirically”.

The OpenAI work replicated it in a completely different environment, and at that point I was sure it was real.

Sometimes people ask me why I was so surprised by it. Prior work like Banko and Brill and the unreasonable effectiveness of data argued for more data. ML theory had similar models for toy problems, eg coin flips.

At the time I thought deep learning was supposed to be complex. Speech and language datasets seemed much more complex than toy problems. Optimization of deep transformers was complex.

The idea that it was possible for the whole thing to be governed by a 3 term equation seemed too simple. The implication was that it was simple to manufacture intelligence.

Ten years later, I still think it is still the most interesting observation I have seen. We are still learning what it looks like to live in a world where it is possible to manufacture intelligence.

He started with tinkercad and thingiverse.

I tried basic elegoo and bambu printers.

He can’t read very well but he likes dragging shapes around on a tablet.

He would ask me to find shapes using the search engines then he mixes them together or reshapes them.

I would add them to his history.

This is why I was surprised to hear about 3D printed guns. I was quite sure there wasn’t anything like that in the history.

It was a good discussion topic about why adults get so bothered by things that look like guns.

My kindergartner has a 3D printer.

I got a call from the school principal. She said “another parent called and said your son 3D printed a gun and brought it to school”.

I looked at the print history. It was a tiny toy mandalorian figurine holding a blaster pistol in his hand.

I bought my son a bigger 3D printer and told him to stop playing with that boy.

Usually breakthroughs in computing lead to more usage of computing, not less.

This is why I use a router to send my own IP to my own models, and general information to Claude.

https://split-brain-ui.scalarxlm.com/docs/clients

I expect Claude to train on my general tokens. I train my own model on my IP related tokens.

This weekend I was reading this paper on programming the Cerebras wafer scale engine, https://arxiv.org/html/2405.07898v1 . Data movement is the expensive part of computing, and some algorithms like stencils only require nearest neighbor data movement per cycle. Cerebras wafers have very low energy transfer between neighboring processing elements on the same wafer, so they come up with a language called Tungsten that focuses on this exchange primitive in the kernel programming model.

I thought the challenge of programming 100,000s of cores using a mesh would be interesting so I wrote a simulator, simple compiler, and a few simple kernels for the wafer scale engine using publicly available documents.

I'm used to CUDA. So I asked: "How would you map something like CUDA onto a machine like this?" Well I use something like malloc to allocate global memory, memcpy to move between host and device memory, and a queue of launch thread block launches, but this time, thread blocks can communicate using nearest neighbor send/recv instructions within the same block instead of through shared memory on a streaming multiprocessor. This is inspired by the stencils in Tungsten.

The whole program is made up of a bulk synchronous kernel of many thread blocks.

I think it is interesting because CUDA has some hard limits on thread block sizes, but this mesh perspective lets you grow or shrink the blocks significantly.

Note that some information about cerebras wafer engines like the ISA is not public (as far as I know). In this code, I just guessed what it could be.

So this should not be taken as a faithful or accurate simulation of the wafer scale engine. More like a point on the design space that is similar in that it includes a wafer sized mesh of processing elements.

What I do is route general data to Mythos, and my own IP to a local model.

I expect them to train on their traffic, and I train on mine.

I don’t get it. That’s what I am using.

Think about the worst enterprise SaaS apps you have used…

What I tell my team to do is to drop using so many cloud saas apps, and build more themselves using LLMs.

I’m not planning on firing people, but I am planning on building more, using more tokens, and less app subscriptions.

One aspect of building that doesn’t erode is human values.

LLMs don’t create software with zero direction and although I do have 12 agents building constantly, I run out of attention to increase that to 100.

It’s about enterprises who care about supply chain risk and having a throat to choke if they have a problem.

Here’s a real example.

I’m in a design meeting talking about a model use case. We have a question about the data pipeline or the prompt format that would benefit from knowing about how the model was trained. The enterprise team lead calls the dev tech engineer from the company who produced the model. He is already in the office and walks into the meeting to answer the question.

There is demand for US open models.

Instead of move to duck duck go I just stopped using search

How far can a pure mercenary culture get?

Don’t put it past Dario to buy spaceX

I’m seeing founders being encouraged to run their business with AI and cut out the etc etc

Smart move

“We hold a meeting to talk about the meetings, and another to plan the meetings about the meetings.“

I dropped scrum last year.

I used to think evil killer robot discussions among AI researchers was an idea based in Hollywood, not science.

Then I realized how effective the fear was at fundraising...

I originally thought evil killer robots discussions in AI labs was an idea out of Hollywood.

Then I saw how effective it was at raising money.

I personally don't mind letting Claude write about work.

You could spend 80% doing the work and 20% writing about it, or 99% doing the work and 1% copy-pasting Claude's writeup about it into a blog.

There is nothing wrong with writing if you are into it, and yes you can probably do better than Claude, but I can related to engineers who just want to build.

One of my lessons in using different accelerators, whether they be different NVIDIA versions, or GPU->TPU, etc is that someone needs to do this work of indexing, partitioning, mapping, scheduling, and benchmarking. That work is labor intensive.

In this case, google has already done it, and that will be true for high resourced accelerator companies like Google working with the most popular operations like attention.

As long as you use those operations, you are okay. But if you do something different, you need to be prepared to do all of this yourself.

Results as good as Qwen has been posting would seem to trigger a power struggle.

I think companies that don’t navigate these correctly eventually lose.

It was inevitable.

This is why I like Dario as a CEO - he has a system of ethics that is not jus about who writes the largest check.

You may not agree with it, but I appreciate that it exists.

I know the frontier “labs” are holding back publications.

I don’t think it will last among researchers who think beyond production LLMs

I remember it being less profitable than graphics for a long time.

It did make money that would be interesting to a startup, but not to a public company.

I remember it differently. CUDA was built with the intention of finding/enabling something like deep learning. I thought it was unrealistic too and took it on faith in people more experienced than me, until I saw deep learning work.

Some of the near misses I remember included bitcoin. Many of the other attempts didn't ever see the light of day.

Luck in english often means success by chance rather than one's own efforts or abilities. I don't think that characterizes CUDA. I think it was eventual success in the face of extreme difficulty, many failures, and sacrifices. In hindsight, I'm still surprised that Jensen kept funding it as long as he did. I've never met a leader since who I think would have done that.

Most people don't appreciate how many dead end applications NVIDIA explored before finding deep learning. It took a very long time, and it wasn't luck.