Comment by inputcoffee | Hacker News Reader

Comment by inputcoffee | Hacker News Reader

There are so many little details to remember when you implement a Neural Network from "scratch". Or, I suppose, even if you do not.

You know what would be a great contribution? An extensive set of unit tests, or even just problems with solutions. That way people can write their own implementations and test them. And even if a person were to implement the net in Pytorch of Tensorflow, they could test the work.

So there would be a matrix of weights, and a vector of input nodes, and the "answer" would be the output vector. Then there would be another "answer" which is the output with a particular activation function, and so on.

This library would just be there so people who are doing their own implementation can test their work.

As I said, a unit test would work too but then it would have to be language specific. Just the matrix and answer would be language agnostic.

For people who think: can't you just make up an example yourself using a sheet of paper or in excel.

Yes, for most purposes this is fine, but if you forget one little implementation detail of a three layer network with a ReLu, you really want an external way to check that.

That would be a great contribution but unfortunately I think not an easy one to provide. A few issues

1. Most BLAS libraries aren't deterministic. Floating point numbers aren't in general associative, (a + b) + c =/= a + (b + c). However most BLAS libraries will happily shuffle the order of operations around if it saves some CPU cycles, I've seen this issue cause huge differences in behavior between different CPUs before, although I think that was a pathological case. So if you want to do a test like this you really need to have everyone on the same page an installing a deterministic BLAS.

2. Training neural networks is stochastic I know you are talking about just the forward pass but I think it would be interesting to have these tests for training your network as well. Naively this sounds like it might be easy to fix (just provide a seed!), but gets very tricky when you want something to output the same code over multiple libraries, operating systems, hardware etc. Multithreading code, putting stuff on the GPU or even the cloud add even more difficulty.

Built-in testing would be fantastic. Being able to tell if you designed a model wrong or just made an error setting up the code would reduce frustration a ton!

If you're implementing your own derivatives, you should probably use numerical methods to check that the gradient is computed correctly, and pytorch comes with tools to help with that https://pytorch.org/docs/stable/autograd.html#numerical-grad...

If you're reimplementing something for learning purposes, you can just compare the output you get with the existing implementation.

But as soon as you're trying to do something novel, no automatic test can tell you whether the model architecture or your implementation of it is at fault when you don't get the results you'd like, because there's nothing you could use for comparison.

Reply on news.ycombinator.com