Tinn: A tiny neural network library written in C99 | Hacker News Reader

Tinn: A tiny neural network library written in C99

github.com

189 pointsflipnotic58 comments

kann [0] is a similarly small neural network library written in pure C. It's quite fast for CPU-only, but only applicable for rather small problems.

[0] https://github.com/attractivechaos/kann

It is fair to say KANN "only applicable for rather small problems". However, KANN is not so "similar" to Tinn. Tinn only implements MLP with a single hidden layer. KANN supports a lot more features such as 1D/2D convolution, RNN/LSTM, weight sharing and mini-batching. It is also more optimized.

You're absolutely right that kann is more optimized and full-featured. I thought to mention your library because it is also a very portable neural network library written in C.

kann is a stunning piece of engineering, and your code is magnificent, and that is precisely why I mentioned it. I apologize, I did not clearly express my intent.

I just meant to add some clarification. I actually do appreciate that you mentioned it. Thank you!

Cranium is another alternative that uses BLAS for efficient matrix operations.

https://github.com/100/Cranium

Just curious, is there a reason for passing around the Tinn struct by-value? Will it get optimized?

It’s only a few bytes, pointers to buffers

It's commonly optimized out, yes. Rule of thumb is typically to pass pointers to a struct if it gets fairly large and complex. Even then, you should really profile to see that the becomes a bottleneck, compilers inline aggressively and a pointer might defeat some other optimizations down the call chain.

It might get optimized by inlining, these functions are pretty small and would make good candidates.

Passing small structs by value isn't as expensive as it once was, modern ABIs pass the initial fields in registers. However, this struct does have ten pointer/integer fields, and I'm not aware of a mainstream ABI that allows that many. x86-64 allows up to six on Linux, IIRC. A quick check of the AArch64 ABI says it takes up to eight. So yeah, there will be stack traffic involved if these function calls are not inlined.

I'm curious too. Even if it will be optimized isn't it still be better to pass-by-reference?

In C?

I mean pointer.

It might be easier to answer if you explained why you think it would be better.

There may be cases where by-reference (yes, that's an absolutely acceptable thing to say, even in C) is more efficient: If the struct is a kind of "this pointer" (as in object-oriented programming) that is passed to a bunch of functions but relatively rarely used, then passing it by reference causes much less register pressure. Passing it by value would mean occupying a lot of registers with values that are probably not accessed.

On the other hand, if you expect the callee to do actual computation on all or most of the fields of the struct you pass in, passing it by value should probably be the way to go. This way, the values to be processed are already in registers and don't have to be stored to the stack by the caller and loaded back from the stack by the callee.

There is no general answer that is uniformly best, it depends on the characteristics of the actual program and their interactions with the compiler's optimizations. With inlining in particular, which should often allow the compiler to generate the same code for both versions.

> There may be cases where by-reference (yes, that's an absolutely acceptable thing to say, even in C)

I don't want to be nit-picking. But I would like to know why "by-reference" is an acceptable thing to say.

When I learnt C, C++, Java, etc. I was repeatedly told that pass by reference exists only in C++. In C, a pointer is passed by value. In Java, a reference is passed by value.

Doesn't calling "pass pointer by value" as "pass by reference" lead to confusion and distortion of the true meaning of "by reference"?

What do you do when you want to find the value a pointer points to? You dereference it of course!

C++ and Java have their own specific language-lawyer definitions of the word reference, but outside of those languages, "reference" is a catch-all term that includes pointers, handles, and indexes.

C implements the concept 'pass by reference' using the mechanism of 'pointer passed by value'.

You have no problem using the terms 'loop' or 'recurse' to describe C code, right? But these high-level ideas are implemented without using those terms in the code. Same with 'pass by reference'.

I would expect it would depend on what the function actually did with the data.

Hi, I'm glouw.

Structs on the stack I pass by value. Structs on the heap I pass by pointer. It's a mental thing that I find helps with large codebases.

Unrelated but thanks for making this! The code is clean and easy to follow.

- a student trying to get a better grasp on neural networks

genann is also a similar small neural network library written in C.

https://github.com/codeplea/genann

Single core? No AVX? What's the point of writing it in C?

It's a demo, like https://github.com/rswier/c4.

Minimal dependencies; people are starting to get fed up with having to bootstrap and/or compile the planet just to be able to run some software. The situation has gotten completely out of hand, especially if one is not on GNU/Linux.

This does nor preclude the use of threads or SIMD.

Essentially this library is a demo toy that will need a lot to work to make useful in real conditions.

That's exactly the point.

> Essentially this library is a demo toy

It's 200 lines of C, what were you expecting?

Generally when I see links to github projects on HN I expect something useful or extraordinary and this looks like my Neural Networks 101 assignment, only far more polished. But people seem to like it(over 400 stars right now) so maybe I am missing something.

People like it because it does what it says on the tin without any further dependencies and having to pull in the entire planet, as with the respective node.js binaries.

So... if its usefulness is equal to something you've made, AND it's available to the public, AND it's more polished... why exactly are you shitposting about it?

Let people enjoy things.

I just pointed out that this is a very basic code. It's definitely not something I would use in any real life scenario(see other posts itt). And if you want to learn about NN you'd be better off googling something like 'backpropagation example' as this project does is virtually not documented. I personally don't find it useful and it takes up the space that can be filled with more interesting content. How is that 'shitposting'?

It's a refreshingly clean, tiny library without dependencies that can do some cool stuff. Let people like things. Stop shitposting.

Thankyou for the kind words.

It's not shitposting.

One of the core requirements of modern deep neural networks is performance. The high accuracy you get from very fast training over large amounts of data is the only reason to look at neural networks over very fast, very accurate systems like XGBoost.

If something doesn't deliver that, then it's worth pointing out that it is inadequate for anything but toy tasks.

It's not really worth pointing this out when the thing in question is a 200 line C library. People won't be expecting it to save the world and/or butter their toast.

+1 also these tiny libraries are great for learning the basics. All of the code has actual meaning behind it and it's super easy to get it up and running.

No. You definitely don't want to look at C code when you want to learn NN basics.

Please make your point without gratuitous inflammation like “shitposting”. Its effect on the threads should be apparent, and we're trying for better here.

Lucky you to have a neural networks 101! My CS degree is now 12 years old, and I crave simple examples like this one. Hence the upvote.

There is a ton of much better tutorials online (e.g. [1]) if you want a NN 101 course.

[1] http://neuralnetworksanddeeplearning.com/chap1.html#implemen...

Expected about 100 lines more for actual performance. ;)

When I'm looking for libraries to use on my Cortex M0 and M3 embedded devices, I prefer small code size to fast execution (as long as it's fast enough) and don't have SIMD and no use for threads. Sometimes size does matter greatly.

The included test script demonstrates that it can learn to recognise the semeion digits with reasonable accuracy in under a second. I've tested it on the pendigits data (8 points to describe a digit) and it can learn to recognise those with 93% accuracy in 1.5 seconds. What more do you want from 200 lines of code?

Writing clean C loops are a good way of writing vectorized code.

Checking the vectorization report of icc 2018 for fprop() shows that the loops got inlined, unrolled and vectorized (with AVX512 and AVX2 instructions using XMM/YMM registers). I doubt that doing it by hand would be faster, but it would make the code a lot more complex.

You have a point on the parallelization part though, it can be cleanly parallelized with a few OpenMP Pragmas.

A thing of beauty happens when you write simple code in C and C++: the compiler has a field-day optimizing ad-nauseum. look at all those zmm(AVX-512) registers being used if your architecture supports it (on top of all the inlining and unrolling)!

godbolt link of the source + assembly -- all I did was combine the header, source and test files:

https://godbolt.org/#z:OYLghAFBqd5QCxAYwPYBMCmBRdBLAF1QCcAaP...

What exactly is the output of the example in `test.c`? I.e. what do the numbers mean? The error in the prediction I guess, but the prediction of what? What is the first entry in the semion data set? a "0"?

No kidding. Does the output mean "hot dog" or "not hot dog"?

The first row of output is the vector of numerals 0-9 where 1.0 indicates "it was this digit."

The second row is the probability of each given the input.

e.g. This output means it's 98.6% certain that the input represents the digit 7.

    0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 1.000000 0.000000 0.000000 
    0.000009 0.000005 0.000000 0.000000 0.003899 0.012933 0.000436 0.986013 0.000005 0.000000

To the author: in your github topics, "propogation" -> "propagation".

Thankyou, my friend ;)

How about in 12 kilobytes of assembly? More featureful implementation here: https://github.com/dfouhey/caffe64

This is a lot more impressive! I'm puzzled by this bit in FAQ though:

Q. It seems that it only does 1D feature vectors?

A. Convolution and other inductive biases are only necessary when you have small data.

That page is likely not completely serious. They also write that compilers are for losers.

> That page is likely not completely serious.

The third line of the readme says "We are finally at our April 1 Release (v4.1.2018)." The date may be relevant. ;)

“Installing a new neural network library is typically a tremendous pain because of onerous dependencies, python version hell, complicated Makefiles, and massive code bloat.”

Finally people have started speaking up and doing something about it! May we see more of such mentality in the future, and computer industry will be a better place to work!

Developers still underestimate how much code bloat and dependency proliferation harm security and administration. Static linking encourages even more bloat by vendorizing dependencies.

Please make your applications modular. Optional dependencies are there for this reason.

Most good packaging tools support them.

Reply on news.ycombinator.com