Hacker News Reader

Comment by dragandj | Hacker News Reader

Let me chip in with some self-promotion.

This book explains and executes every single line of code interactively, from low level operations to high-level networks that do everything automatically. The code is built on the state of the art performance operations of oneDNN (Intel, CPU) and cuDNN (CUDA, GPU). Very concise readable and understandable by humans.

https://aiprobook.com/deep-learning-for-programmers/

Here's the open source library built throughout the book:

https://github.com/uncomplicate/deep-diamond

Some chapters from the beginning of the book are available on my blog, as a tutorial series:

https://dragan.rocks

View on HN

Machine Learning in Clojure reminds me of Yann LeCun’s ML course from 2010, where we used an adorable language called Lush:

http://lush.sourceforge.net/

which I suppose can best be described as Lisp and Python having a baby. It was immense fun to code neural networks from scratch in it. I hope Clojure can find a bigger place in the world of ML.

Another student of LeCun from NYU here. Can attest that lush is adorable. For example:

For high performing parts of your code, a subset of lush would generate C code and compile them. I imagined that this is what it was like to write the first version of C++, the one that generated C code.

In those days there was a lovely LuaJIT based tensor manipulation language torch7 [1,2] developed by Leon Bottou. It later became basis for PyTorch. I still believe that Lua in general and LuaJIT in particular are much superior to Python for Deep Learning.

[1] http://torch.ch/

[2] https://github.com/torch/torch7

I'd actually love that material in C++/CUDA.

If only C++ supported interactive REPL and the rest of Clojure/Lisp goodies, that might be possible. However, the code is CLOSELY related to the actual CUDA/C++ api. It's a lot simpler, concise, and everything, but I explain everything so that you can use the relevant parts with cuDNN and DNNL APIs in any language that you're most proficient in.

Does Cling C++ interpreter do what you want? https://github.com/root-project/cling

Clojure does what I want.

Perhaps Cling can do something, or not, but I guess that's up to people who prefer Cling to find out and utilize.

Have you used it? When I last tried Cling not that long ago, it wasn't even alpha quality software and given that it has been around for a while, my default assumption would be that this hasn't suddenly improved.

it's used by ROOT at CERN, so it's unlikely it's "alpha quality software" since it's being used for production science at scale.

Thanks! That's helpful to know. I have no experience with Clojure/Lisp but a fair amount with C/C++ (minor in CUDA).

Please read a few of tutorials from my blog. Most programmers in your situation told me that they had no problems following it; it only gradually introduces advanced Clojure concepts, and code snippets are usually extremely short + completely executable interactively as-is.

This is a great site. A few pieces of unsolicited feedback from a marketing perspective:

* Always have the call to action repeated at the bottom of the page; you did a great job having it above the fold, but I got to the end and had to scroll back up to express intent. That's a flowbreaking design.

* This is purely stylistic, but I strongly prefer the first letter of each line to be capitalized. I find the stylistic inconsistency around capitalization offputting.

* You have pretty nicely made docs, consider making the link to view the sample chapters a clickable picture of the first diagram-page in your chapter.

Thanks for making this!

Thanks! I have this slated in my to-do tabs (which I actually do lol). Just looking through it looks like you did a lot of work and took a lot of time for this so just wanted to say thanks.

"Deep Learning in Clojure with Fewer Parentheses than Keras and Python"

Love it! :D What better way to define a neural network in code than an S-expression?

I am not sure if I am that enthusiastic. The problem with Lisp is not a number of parenthesis but where they are and what is their role. In c-like languages parenthesis help parser compiler but they also help humans to read the code. In case of Lisp they are just for the sake of the parser. Let's look on the code:

Python:

  model = Sequential()
  model.add(Conv2D(32, kernel_size=(3, 3),
                 activation='relu',
                 input_shape=(28, 28, 1)))

Clojure:

  (defonce net-bp
    (network (desc [128 1 28 28] :float :nchw)

Which one is more readable? looking on the Clojure code I see 128 1 28 28 thrown on me, without digging in the documentation I have no idea what's happening.

> Which one is more readable?

Both are equally readable to me.

Now, granted 128 1 28 28 can be difficult to understand without documentation but that is not due to Lisp's fully parenthesized prefix notation. The Clojure code would also look equally readable if it had used keyword arguments.

Are you sure you are not confounding familiarity with readability? With Lisp, after a while, the parentheses become invisible to the programmer.

But that's due to named parameters vs. positioning, not parenthesis; I can mirror your argument with:

Python:

  model = Sequential()
  model.add(Conv2D(32, (3, 3),'relu', (28, 28, 1)))

Clojure:

  (defonce net-bp
    (network (desc :input-shape [128 1 28 28]
                   :type :float
                   :something :nchw)

It all depends on how desc is defined.

Disclaimer: no idea what those functions mean, I only made syntactic changes.

I really enjoy your work. I bought your book recently and appreciate your approach of building an understanding of the library based on "first principles". Really appreciate this performant and elegant option for working with deep-learning in Clojure - Thank you!

Thanks!

Concise isn’t always better.

You’re throwing alway all the names of the arguments and using arbitrary words like “conv” to represent operations.

This is typical bad clojure in my experience; write once, forget wtf the magic was, throw away and rewrite it again later.

Clojure doesn’t have to be incomprehensible arcane magic that does everything in 10 lines.

The more complex the code, the more important it is that what you do is clear and clearly documented.

Don’t write a 1 line regex to solve a complicated problem; it’s the wrong tool for that job, no matter how smart your substring matches are.

You don’t win a prize for making unmaintainable code.

I similarly think the goal of being burning my concise in ML code is deeply misguided.

How is "conv" arbitrary? There is a function object that represents a convolutional layer in the network. It is bound to two symbols (because why not). You can either use "convolution" if you prefer full names, or "conv" if you prefer shorter. It doesn't represent the operation, but the layer. There are functions (with longer names) representing the convolution operation, which follow cuDNN and DNNL naming schemes.

Regarding the magic, I believe you haven't read my writings related to this. Exactly the opposite - there is no magic other than usual Clojure-fu, which I explain in a layered way.

But it's difficult to exactly reply to your critique, because you haven't given any example of an approach that would be good Clojure. Ok, give me an example of how you would do it in a comprehensible way (if what I provide is incomprehensible). You don't have to actually implement it. Show a non-working alternative. How would it look like?

Concision is a style choice to be used with care. Spending screen space on additional characters and descriptions detracts from the ability to fit more logic on the screen at once and grok the larger flow. Splashing symbolic alphabet soup into your IDE in the name of concision isn't usually a good idea, but naming something "conv" in the immediate local context of a convolutional layer doesn't seem so bad.

Does 'convo' refer to a 2D convolution or a 1D convolution? Given the large number [1] of arguments that a convolution can take, which ones are being specified? I can probably guess, since only 2 are given, but if there were more, which order would they be in and which would refer to which?

The code is on github [2, 3] see for yourself if you think it's more or less obvious than the python equivalent of a 'trivial' network.

I would say 'conv2d' is probably reasonably standard in meaning; I refer to 'convo' as arbitrary, because it is. Either (ideally) avoid abbreviations, or use standard ones.

[1] - https://pytorch.org/docs/stable/generated/torch.nn.Conv2d.ht...

[1] - https://github.com/uncomplicate/deep-diamond/blob/master/tes...

[2] - https://github.com/uncomplicate/deep-diamond/blob/master/tes...

I know nothing about neural networks, but it looks to me like convo is smart enough to create the convolution of the correct dimension based on the arguments. Where as Keras seem to force you to use a different constructor for different dimensions. That explains why it's called convo, since it creates convolutions of any dimension.

Also, most options are provided as a map to convo as well it looks like, so you'd have similar named arguments for convo once you get to defining optional things like padding and strides.

convo is smart enough to cover 1D, 2D, 3D, and any other convolution layer that the backend can support. You only need to specify the data that it can't figure out, but you can specify more if you want.

Reply on news.ycombinator.com