Hacker News Reader

Comment by modeless | Hacker News Reader

I think many people don't appreciate just how simple state-of-the-art neural net techniques are. It's great to have an illustration of just how little code you need to get the results that have amazed people.

You could say that it relies on PyTorch which is a lot of code, but most of the complexity comes from the need to do GPU/TPU/SIMD acceleration. A truly from-scratch CPU-only implementation in C would still not be a large amount of code.

View on HN

As a developer with interested in neural-net based ML, my eyes start to glaze over a bit when I see so much crazy-looking numpy operations even just to get a trivial representation of data.

I guess it's just something I have to get used to, but I wish there was an interface to do the same nd-array based logic that was designed more for developers like me rather than data scientists who perform surgery on 50d arrays all day long.

I agree; check out NamedTensor for an attempt to fix this: http://nlp.seas.harvard.edu/NamedTensor

Don’t look at code, because it bakes in so many derived values and simplifications that it’s hard to recover the original ideas. If you can find the original math, and walk through the derivatives (seeing zeros crop up and get simplified away), it starts to make a lot more sense.

Have you checked out numba? You can use regular Python for loops and numpy calls and it will compile it. Instead of numpy vectorization mess.

Good luck with anything of non trivial types. Numba is awesome but also quite finicky.

Sorry if this sounds condescending but just learn linear algebra and numpy. Linear algebra really is a minimum requirement if you want to get anywhere near ML. College freshmen can learn linear algebra and numpy and so can you. You'll only get even more lost and frustrated if you try to go into it thinking in terms of for loops.

It's kind of like saying Redis is just an dict. The optimizations in the entire software/hardware stack is what makes these innovations possible. A C implementation would not work in reality for large models.

Here is a C implementation of GPT-2. https://bellard.org/nncp/gpt2tc.html

I don't disagree that hardware acceleration is key in enabling these models, but I still find it interesting how simple the core techniques are.

Is there anything Bellard hasn’t done?

He didn’t release blueprints of cheap homemade 1MW fusion reactor yet, but otherwise - yep, it seems that he covered everything else.

he didn't rewrite everything in rust.

Once he picks up Rust, he'll become a 10x programmer.

What a loss. To be downgraded that much.

That's not really a C implementation of GPT-2 since it cannot be used to do the thing everyone cares about: self-supervised learning from text. In fact, it doesn't even use the weights in the same way GPT-2 does, so it's not clear how close it is to GPT-2's inference mode. The source isn't even on the page.

Hm, I notice the source code is missing? I did I overlook it

This is very cool, thanks for sharing! From the readme (https://bellard.org/nncp/readme-gpt2tc.txt), the program benchmarks very comparably to CMIX, which is the top algorithm on the Large Text Compression Benchmark (http://mattmahoney.net/dc/text.html). I'm guessing that any GPT implementation would be ineligible for the benchmark because of its file size but impressive nonetheless.

> A C implementation

You mean CPU implementation. C language isn't the culprit here, and low-level parts of ML frameworks are normally written in C or C++ as well and compiled with nvcc or equivalent.

It's simple in the same way Conway's Game of Life is simple.

It really depends on what level allow yourself to start.

If you hand derive all equations in your program you are going to have a hard time ... but automatic differentiation (ad) and existing linear algebra libraries (for gradient descent/ADAM) make the the job pretty easy.

I recently dabbled in automatic (song-)mixing ... reimplemented the code of some open audio plugins in a way that they were auto-derivable and "trained" the whole thing on some multitracks with reference mixes. Sadly that didn't resulted in a good mix ... but in theory it should work.

What's your "simple deep learning" story?

Karpathy is also very good at making complex architectures to very simple.

Reply on news.ycombinator.com