Comment by fartcannon | Hacker News Reader

Comment by fartcannon | Hacker News Reader

Excellent! Very kind of you to do this. Are you accepting comments? If not, ignore the following. :)

As someone who learned how to program from trial and error via tutorials on the internet (some of the people who are going to read your book are people like me), I just have one comment:

Try not leaning on libraries in your tutorials.

I know it sounds insane to suggest you not use numpy in any kind of ML tutorial, but libraries like numpy used to make my new programmer's eyes glaze over. The fact that those imports are black boxes and can do literally anything used to make my noob mind overload. And I'd develop a blind spot for them. Before I could read code that exploited numpy's power, I had to work with numpy for a while to gain an intuition for it. New people don't have that. This isn't just towards numpy, but for all libraries in tutorials. If it's at all possible (I know it's not always possible in a reasonable way), make a super simple example of what you're going to use the library to do before you replace it with the library.

As mentioned elsewhere in this thread, Sentdex is actually really good at teaching new developers. I think this is because he often starts without numpy. For example in NNFS videos, he starts with just lists. He gets to numpy eventually, but knowing what he replaced with numpy helps make things more clear.

Just a thought.

Thanks so much for your feedback. Definitely open to comments!

I agree 100% that any use of packages can be intimidating for newbies. I experimented at first with creating the models without using numpy and I thought that it actually made it less clear rather than more clear. It's obviously a tradeoff--you see where everything comes from (rather than np.mysterious_function()) but you take 5 lines of code to do the same thing that a single numpy command could accomplish. I felt in the end that it distracted from the real purpose of the code, which is to demonstrate how the model works.

Do you think a compromise would be to add a section to the appendix introducing numpy? Introducing arrays, random instantiation, stuff like that? Otherwise I might consider adding a no-numpy version in the future.

Thanks so much for your feedback!

I'd love to include the book to our company internal learning resources. Can you include an official license by any chance? Thank you

I hadn't even considered licensing it. Want to email me and we can talk? My email is dafrdman@gmail.com. That said, you're welcome to use it (though my lawyer father suggests I say that this "verbal contract" is revocable and non-exclusive).

Perhaps you could reduce the set of numpy functions used in your code to a minimal set (exp, sum, max, min, etc.) and then build fancier functions up from there. This affords you the speed and conciseness of using numpy arrays while limiting the abstractions that could obfuscate the inner workings of some of the fancier functions you might use (e.g. softmax).

That's a good point. If you had to explain how everything works without the libraries, you'd probably end up writing a book on how pandas and numpy works, not how ML works.

An appendix is a great idea!

Good ideas. I think I'll try to add an appendix, minimize the number of numpy functions used, and explain any of the weird ones that are real time savers. Thanks for all your thought.

I think there is a balance to be struck. You should totally use numpy for the arrays and basic math applications. But say on the first example you use `self.X.T` what does `.T` even do? Not asking you to go into all the details, just more comments saying this transposes the array, see numpy docs <link>. It will ease people into the library if they are unfamiliar with it. You do have some good ones like `column of ones` already, but more of those kinds of things.

I would also avoid using pandas if at all possible. Its just another thing people have to learn if they are unfamiliar.

I definitely agree. I should add more comments explaining what things like .T does--it's not that it's hard to grasp, but it might turn away newbies. Thanks for the suggestion!

Pandas is only used in the "code" sections, which use packages like scikit-learn anyway

I have a different view :)

I feel this book is the ideal companion for intermediate to advanced Python developers/data scientists. People who know pandas pretty well, and so are comfortable with numpy array operations too. They have probably used scikit-learn's fit/predict API in a black-box way, but have never quite had time to look at the code underneath.

I love "ML from Scratch" because it leverages my math and Python/numpy knowledge (even if some of it is rusty and half-forgotten) to show me high quality, well-commented, mathematically rigorously explanations of how I might have implemented the types of algorithms I use from sklearn. While the maths is formal, it is totally straightforward in its presentation.

It will definitely be my bedtime reading for the next few nights.

I disagree slightly. I think numpy (and scipy is so central to do anything mathematical in python (actually you could say python is just the glue for these libraries) that maybe one should spend a chapter or two on numpy instead of not using it.

I agree with you in theory. I really hate that the fast.ai course uses their own fast.ai library.

Numpy is a bit different though as it “fixes” inefficiencies in Python’s typing system. A list of Python integers is actually a list of C structs so none of the values are sequential in memory. A numpy array of integers is sequential in memory and so the performance gains are massive.

It’s not so much a library as it as a way to access efficient data types/structures. I think any mathematical programming in python should start with numpy.

If the author had used Pandas extensively I would agree with you completely.

This was exactly what my professor used to say while we were learning programming. Our thought process should be at building the program, to construct a progressive logic train of thoughts that is robust enough to avoid basic errors. Using packages defeats that purpose. Once we have a minimal working solution, we can then worry about efficiency, scalability and other benchmarking parameters to judge how far is the first try from the best one.

While developing something though, it could be appropriate to use existing and better solutions.

I wrote this article you might like: https://github.com/jeremyong/cpp_nn_in_a_weekend/blob/master...

No dependencies. Just straight C++. The code is heavily annotated, doesn't use any matrix libraries or anything, and gets 92% accuracy or something without any fancy techniques on the MNIST handwritten digits database.

I wholeheartedly agree with this. As a software developer without a strong math education, any time I open a "from scratch" ml learning resource, I immediately close it because I don't have an understanding of the math happening in the black box of numpy.

Thank you for the Sentdex recommendation.

Can you elaborate? As an experienced numpy user, I don't really have an intuitive feel for what you describe. Numpy isn't really math it's mostly an array library for splitting, merging, reshaping arrays, swapping dimensions etc. and applying operators on all elements at once.

Outside of the things under np.linalg or np.fft, I cannot think of much really difficult math in numpy.

To grok numpy what you really need is a good mental model of the multidimensional array, reshapes, reductions, transpose and so on. But this isn't really math, it's more data wrangling and kinda boring and tedious "data chores" we need to do before we get to the interesting parts of the job.

Reply on news.ycombinator.com