34 karma · joined January 10, 2024
My interests are: Lisp/Scheme, C code, startups, HW.
I urge you not to take these opinions as facts. Originality is admirable, but it is not "your potential", "proof of great thoughts", or "the most impressive thing you can do".
The answer to the question: What to do? is not "Make new things", but rather begins with a simple question: In what context?
The idea of dividing people into two categories: 1) those who "take care of people and the world", and those who 2) "make good new things", is harmful.
Most Lisp/Scheme implementations offer tracing. Amazingly, the SICP book even shows tracing as a means of analyzing whether a recursive function is tail-recursive.
What I'm curious about is the Lisp history of tracing. Does anyone know which implementations were the first to provide tracing functionality?
Also when it comes to macros, does that include `syntax-rules` or `syntax-case` style macros, where the latter are much more powerful?
While an embedded Scheme-like language is incredibly useful, at some point I feel as if you would simply have to include these features, and to that end it would just be Scheme reinvented.
So? Just stop learning. Close that book, video, course. Go make something people want. Or you want. What would you like to make?
But let me stick with some basics. So, I can write and compile an x86 test.c program. Then, I use your extension and undo the linking. Then, I use the results to link again into a new executable? Are the executables identical? When does it break?
How much of a task is it to make it a standalone program? What about x64 support?
If removing weights improves some metrics, that may be a clue that the model is not optimal in some sense.
In practice, perhaps a C program that acts as a validation test. The source code of this C program is not publicly available. Only the binary is distributed. Let us name the binary ctestbox.
When ctestbox is run, it creates a multiplicity of new text or binary files. Each of these is like a unit test.
Consider a tool that decompiles a binary. Given ctestbox, this tool should make a.out which when run, ideally creates identical text or binary files. Now you simply count the number of identical files as a metric.
My prediction is that this percentage will increase with time. It would be interesting to construct data for this metric.
It is important to define the limitations of using LLMs for this endeavor. I would like to emphasize your subtle point. The compiler used for the original binary may not be the same as the one you use. The probability of this increases with time, as compilers improve or the platform on which the binary runs becomes obsolete. This is a problem for validation, as in you cannot directly compare original assembly code with assembly after compiling C code (that came from decompiling).
Perhaps assembly routines could be given a likelihood, as in how sure the LLM is that some C code maps to assembly. Then, routines with hand-coded assembly would have a lower likelihood.
What is unclear to me is: why did the authors fine-tune the DeepSeek-Coder model? Can you train an LLM from zero with a similar dataset? How big does the LLM need to be? Can it run locally?
What I would like to do is to use Lisp to solve complex problems. The problem is that abstractions are not often efficient.
For example, consider computing the calculus product rule. An S-expression abstracts the function: (lambda (u v) (* u v). With metaprogramming, this becomes a new S-expression (lambda (u v) (+ (* (d u) v) (* u (d v)))), where (d u) and (d v) further compute the derivative. To optimize this further, you need implementation dependent code and replace abstractions of S-expressions with native machine code. This is where I feel that the troubles with Lisp start.
I'm curious, why do you believe int8 will dominate?
On the other hand, esp-nn seems to be code for the xtensa instruction set. I briefly overviewed the instructions. They seem optimized for DSP rather than ML applications. Searching for SIMD returned no arithmetic instructions. Searching for parallel returned instructions for multiply and accumulate. Further, the FPU does not compute any kind of 16-bit floating point numbers.
>ARM has also defined a new set of extensions for NN acceleration Can you provide some more info about this?