gdiamos

I'm not sure why the article dismisses cost.

Let's say X=10% of the GPU area (~75mm^2) is dedicated to FP32 SIMD units. Assume FP64 units are ~2-4x bigger. That would be 150-300mm^2, a huge amount of area that would increase the price per GPU. You may not agree with these assumptions. Feel free to change them. It is an overhead that is replicated per core. Why would gamers want to pay for any features they don't use?

Not to say there isn't market segmentation going on, but FP64 cost is higher for massively parallel processors than it was in the days of high frequency single core CPUs.

I feel like a lot of software engineering problems come out of people who refuse to talk to each other than through comments in VCS.

It makes sense if you are collaborating over IRC, but I feel the need to face palm when people sitting next to each other do it.

What is your preferred way to talk to your team?

No English, only code

Slack

Zoom

In a meeting room

Over lunch

On a walk

One thing I’ve learned over time is that the highest bandwidth way of talking is face to face because you can read body language in addition to words. Video chat is okay, but an artificial and often overly formal setting. Phone is faster than text. Text drops the audio/visual/emotional signal completely. Code is precise but requires reverse engineering intent.

I personally like a walk, and then pair programming a shared screen.

It sounds like lack of security is the biggest feature and risk of this clawd thing.

I also tried using Siri to tell me the weather forcast while I was driving to the park. It asked me to auth into my phone. Then it asked me to approve location access. I guess it was secure but I never figured out what the weather forecast was.

Thankfully it didn't rain on my picnic. Some of the parents there asked me if their investors should be interested in clawd.

Here’s another perspective. Developers aren’t budget approvers in engineering organizations by choice.

If you are a budget approver then your inbox and calendar are full of sales teams.

I find that experience is too distracting to concentrate on writing good code.

It was an interesting read Sara, thanks for sharing it.

I especially agree with your point that scaling laws really killed open research. That's a shame and I personally think we could benefit from more research.

I originally didn't like calling them scaling laws.

In addition to the law part seeming a bit much, I've found that researchers often overemphasize the scale part. If scaling is predictable, then you don't need to do most experiments at very large scale. However, that doesn't seem to stop researchers from starting there.

Once you find something good, and you understand how it scales, then you can pour system resources into it. So I originally thought it would encourage research. I find it sad that it seems to have had the opposite effect.

The iphone is so useful they can probably ride it for a couple more decades at least. I would still buy it.

From a technology or engineering perspective, I have no idea how to work with Apple.

A Dyson sphere brain ?

I personally don’t think the scaling hypothesis is wrong, but it is running up against real limits

What high quality data sources are not already tapped?

Where does the next 1000x flops come from?

Diffusion is favored by current GPUs .

Over time we seem to have a tendency to build models that are well matched to our machines

How do they break ties?

Did I miss something? https://github.com/NVlabs/Fast-dLLM/blob/main/llada/chat.py

That’s inference code, but where is the high perf web server?

Thanks I’ll check it out!

One year later and there is still no inference engine for diffusion LLMs

Students looking for a project to break into AI - please!

It was risky.

From my perspective it was more exciting to the programming systems and compiler community than to the computer architecture community.

Transmeta made a technology bet that dynamic compilation could beat OOO super scalar CPUs in SPEC.

It was wrong, but it was controversial among experts at the time.

I’m glad that they tried it even though it turned out to be wrong. Many of the lessons learned are documented in systems conferences and incorporated into modern designs, ie GPUs.

To me transmeta is a great example of a venture investment. If it would have beaten Intel at SPEC by a margin, it would have dominated the market. Sometimes the only way to get to the bottom of a complex system is to build it.

The same could be said of scaling laws and LLMs. It was theory before Dario, Ilya, OpenAI, et al trained it.

The role of basic research is to get off the beaten path.

LLMs aren’t basic research when they have 1 billion users

What is going on at meta?

Soumith probably knew about Lecun.

I’m taking a second look at my PyTorch stack.

This is the end of an era. Amazing work soumith.

ScalarLM uses tokenformer adaptors by default, which have learnable key/values

https://www.scalarlm.com/blog/tokenformer-a-scalable-transfo...

> An expert human compiler... yes, that would be SufficientlySmart.

> And slow.

Build faster GPUs.

> And expensive.

The more you buy the more you save.

> And prone to error.

Find ways to keep scaling laws bringing down the error.

pip culture

I find it helpful to read a saxpy and GEMM kernel for a new accelerator like this - do they have an example?

Reminds me of the movie memento

This is exactly the intuition that leads to excitement about fine tuning.

However, I personally think that this intuition applies to products and interfaces, not to AI.

Intelligence and learning is general. Intelligence without generalization is memorization, which seems to be less useful in practice.

The problem is that it doesn’t always work and when it does fail it fails silently.

Debugging requires knowing some small detail about your data distribution or how you did gradient clipping which take time and painstakingly detailed experiments to uncover.

I agree this is the main case where it makes sense.

But a recent trend that cut into the cost savings is that foundation model companies have started releasing small models. So you can build a use case with qwen 235B, then shrink down to 30B, or even all the way down to 0.6B if you really want to.

The smaller models lose some accuracy, but some use cases are solvable even by these smaller and much more efficient models.

Users often found it hard to know exactly where the boundaries are.

This is a reason why general purpose models shine. You don’t have to carefully characterize a task and put guard rails around it.

llm.finetune(data) is a leaky abstraction

Read Andrej’s blog that I linked earlier in the thread if you want to understand why.

I think you are saying to go after the very high end of the market.

That’s fair, one market segment of this is sometimes called sovereign compute.

Another common model that I have seen is to become the deepmind for one very large and important customer.

I think this works.

Fair didn’t seem to matter to users who just wanted to build solutions with reasonable time and budget