> With your linked spiral code, I can't even really tell where the algorithm starts and where the setup code ends - which of course might easily be attributed to my unfamiliarity with Spiral ;)
Heh, I've noticed that the setup code tends to come out longer than the actual kernels. It is inside `kernel = cuda`. Spiral is indentation sensitive like Python and F#.
The stuff inside {} is just module creation, think of it like tuples with named fields.
> Do you have any benchmarks for the softmax kernel? If that kernel has optimal performance, it would be quite interesting. If it's sup par, it looks much longer than a simple version.
No, I've yet to actually benchmark it. It really depends on how good of a job NVCC does with the generic sequential reduce kernel. I'll do an in depth analysis when I am done with all the neural network work that I am doing currently which might take a while.
https://github.com/mrakgr/The-Spiral-Language/blob/7ecd30bdf...
You can see how it is implement here for the forward and backward parts.
https://github.com/mrakgr/The-Spiral-Language/blob/7ecd30bdf...
In the actual cost function, it is a bit different since I fuse the forward and the backward parts.
It is a bit ugly because of all the type checking for whether the argument is a dual, but that is all done at compile time.
> https://mikeinnes.github.io/2017/08/24/cudanative.html
This is actually loop unrolling example that I had in mind when I said that Julia needs macros for.
https://github.com/mrakgr/The-Spiral-Language#3-loops-and-ar...
You can in fact get loop unrolling with just standard functions in Spiral. I go into it in the context of this chapter. It is achieved by pattern matching over tuples and recursion.
> If you can make concrete examples of how things can be simpler than this, I'd be delighted to hear them :)
Yes, by replacing meta programming with intensional polymorphism and inlinining guarantees. Also reflection should be done using pattern matching. I think this last one could be taken entirely seriously as it would not involve a replacement of the entire type system.
All the claims in the article about inlining and specialization that make it sound like magic is what in general makes me dubious about languages pretending to be speed kings. Yes, I am aware that GPU kernels do not require optimizers capable of having monads for breakfast and that in the context of GPU programming where they were made they are probably true, but inlining is the sort of thing that matters more the more high level a language is. For very high level languages that desire speed, there isn't much choice but to make them a part of language semantics despite the added burden it puts on the user.
Since we are still at it, I have a question I need to ask.
Recently I've been informed that Julia is capable of GCing GPU memory. If this is fully integrated that would be a major feature which is not possible in say .NET or Racket. I really wanted this in Spiral and could not get it in .NET.
By fully integrated, I mean much like for regular memory for which the GC takes note of the state of the system for when to do collection and defragmentation. If it can only make a thin wrapper with a finalizer (much like in .NET) around an unmanaged resource then it is not a big deal.
Is it fully integrated or is it a wrapper style memory management?
If it is the later, then that is too bad, but I'd suggest to Julia devs that they work on making it fully integrated as it would be a really good feature. Obviously, I can't do it in Spiral as I would need to write my own VM and I have only so many years in my life.