as this paper shows, it sees they can do tower of hanoi as well, up to a certain point that is.
1,929 karma · joined March 28, 2022
Maybe the way forward is in LCM or go JEPA, therwise, as this Apple paper suggests, we will just keep pushing the "pattern matching" further, maybe we get some sort of phase transition at some point or maybe we have to switch architecture, we will see. It could be that things change when we get physical multimodality and real world experience, I dont know.
That is what I am struggling with, it is really easy at the moment to slot LLM and make everything worse. Mainly because its output is coming from torch.multinomial with all kinds of speculative decoding and quantizations and etc.
But I am convinced it is possible, just not the way I am doing it right now, thats why I am spending most of my time studying.
I dream if err, if err dreams me.
I started writing forth few months ago, wrote few interpreters, with jit or with types etc, and its just amazing, I think anyone should do it. TBH I don't think any other exercise has thought me as much about programming as this.
I also notice the "return" of Forth, as it is probably the easiest high level language to make for computers with addressable memory and fetch execute cycle. The parser is just few lines of assembly and of course you can write the parser in the inner interpreter's bytecode, you don't even need assembly :) So hobbyists can just "make it" and make their own tiny operating systems with it. Of course everyone makes their own dialect, but I think thats OK. Things like https://github.com/howerj/lfsr LFSR CPU/VM running Forth, or UXN or duskos/collapseos.
Now you can also use language models to help you onboard into the language, it do some practice programs and rewrite one program in many ways.
So if you are young or old and never tried to Forth, don't miss out, its super fun.
I am sorry, by no means I think it is a trivial token predictor, or a stochastic parrot of some sort. I think it has a world model, and it can do theory of mind to us, but we can not do theory of mind to it. It has planning as visible from the biology of language models paper.
> So again, I disagree about AI-generated tokens having no meaning. But I would agree there is no human connection there
What I argue is that language is uniquely human, and it is how it is because of the human condition. I think we agree more than we disagree. I say that the meaning is 'halved', it is almost as if you are talking to yourself, but the thoughts are coming from the void. This is the sound of one hand slap maybe, a thought that is not your own but it is.
I guess I am saying is that AI is much more like Alien than Artificial, but we read the tokens as if they are deeply human, and it is really hard for people to not think of it as human, purely because it uses language in such profound way.
Popularity is somewhat proxy for 'good', but not always.
Is it going to be useful for language models to train on it? I think so, and I don't mind that. As long as they develop better world models and understand human language better.
The problem I have is with humans reading generated tokens. Human language is shared experience, the evaluation and interpretation of the symbols depend both on the author and the reader (even though many times they are the same entity).
When a person with HPPD says 'The sky is black', when the symbols enter your mind they are superimposed with your experience and their experience to create meaning. (HPPD is a disorder from damaged filters on the visual system, it seems that raw information from the eye sensors are entering the brain, and they can see the inside of their eyes when they look at the sky, so it looks black, as if the whole sky is filled with 'floaters)
When you read AI generated content, you are both the judge and executioner, the symbols mean whatever you want them to mean, they have no author (in the human sense).
So, I want to write for other humans to read :) Even if nobody reads it.
In gpt2 times I used to read gpt generated text a lot, I was working on a game to guess if the text is AI generated or not, and for weeks while I was working on it I had strange dreams. It went away when I stopped consuming tokens, in gpt4 age this does not happen as I am reading hundreds of times more tokens than back then, but I think it is just more subtle.
Now I use AI to generate thousands of lines of code per day, at minimum sometimes now I just blank out when the AI doesnt spit out the code fast enough, I dont know what am I supposed to write, which libraries it is using what is the goal of this whole function etc, as it is not my code, it is foreign and I honestly dont want to be reading it at all.
This week I took the whole week off work and am just coding without AI and in few days the "blank out" is gone. Well, I did use AI to read 300 page docs of st7796s and write barebones spi driver for example, but I treat it almost as an external library, I give it the docs and example driver and it just works, but it is somewhat external to my thought process.
People argue that all fields have evolved, e.g. there are no more blacksmiths, but I argue that the machinists now are much more sophisticated than the ones in the past, pneumatic hammers allow them to work better and faster, as they use the hammer they get better understanding the material they work with, as in the machine does not take away their experience and ability to learn. I always had 2 days per week where I code without any AI, but now I think I have to change the way I code with it.
AI is for sure making me worse, and lazy. And I am not talking about the "career" here, I am talking about my ability to think.
I wrote few days ago about it: https://punkx.org/jackdoe/misery.html
I am very happy that apple moved away from the quest of making the keyboard 0 height.
I have been making more and more time at night to reconnect with the craft.
I think more people in the community might be struggling, so I just wanted to share my way of dealing with it.
Please share how you are dealing with the tokens.
You could argue that those tools in the hands of skilled craftsman will create amazing things faster, but we all know what will happen is absolute flood of AI slop in every entertainment category.
I have been programming in all kinds of languages, from assembly to clojure, but in 25 years I never programmed stack languages, I was kind of scared of them, it wasn't until I read the book and made my own Forth I understood what I was missing. Since then I made few interpreters, with jit, or with types, etc, it was super fun, but most of all it allowed me to see a completely new paradigm of programming, kind of the first time you understand eval/apply from 13th page of the LISP 1.5 Programmer's Manual. A language that writes itself and it is written in itself.
If you are making your own Forth, this Brad Rodriguez's article is also really good [2].
[1]: https://archive.org/details/R.G.LoeligerThreadedInterpretive...
| Name | Semi-private eval | Public eval |
|--------------------------------------|-------------------|-------------|
| Jeremy Berman | 53.6% | 58.5% |
| Akyürek et al. | 47.5% | 62.8% |
| Ryan Greenblatt | 43% | 42% |
| OpenAI o1-preview (pass@1) | 18% | 21% |
| Anthropic Claude 3.5 Sonnet (pass@1) | 14% | 21% |
| OpenAI GPT-4o (pass@1) | 5% | 9% |
| Google Gemini 1.5 (pass@1) | 4.5% | 8% |
thats not the only thing it says, but it does say it:
> The conclusions to be drawn from the points made throughout this paper are reasonably obvious: We need a new language for teaching novices at secondary school and introductory university level.
It also does states that somehow we know how to teach programming, and the discussion is around finding an incremental improvement given the learnings so far.
I am not really sure how teachers get hired, how they are interviewed and assessed, but the system is clearly not working; particularly in subjects that the learning curve is not a curve (like math, or physics or cs)
There are university students now in their 3rd year that paste chatgpt's javascript program into their c# code (also written by chatgpt) and ask me why is everything red and its not "working".
I think it is a problem of designing for the 'average' student, which of course does not exist.
The academia focuses too much on the abstract languages and constructs, but the reality is, the code will run on a man-made digital computer, which has certain properties, it uses memory in a certain way, has a clock, and is discrete, certain things are possible, and others are not. No matter how much we pretend, the equal sign (=) leaks through, and that in itself creates infinite confusion in people who pretend to get it.
I think hyper individualistic approach is the solution. My daughter for example really dislikes scratch and various graphical languages, but she does really well with machine code and assembly and c. When we were working on a chess game, and she said: "WAIT! the queen is just rook + bishop", we went into how can we actually structure our code to express that. The journey is completely non linear. Sometimes we go back, sometimes forward.
Hopefully AI + teacher can help with that. But the truth is, some kids get various concepts, like recursion or references and values, naturally, some never get it. And we are trying to teach them the same thing at the same time for the same duration, and just move on.
SICP is best for some, PDP11 machine code for others, godot or roblox lua for third, and so on. The problem is we don't know which is which :)
PS: Some of the things I made to teach my daughter: https://punkx.org/
-- Carl Jung, The Undiscovered Self
Everything in the average is infinitely reduced, even if you make a chair, a perfect chair for one is torture for another. So we compromise for all.
It is the same with user interface, or education.
Now however I think we can break out of this, with new unique interfaces for each individual, or teaching every child what they are struggling with, be ahead or behind the other children, there is no need to teach quadratic equations to 1 million kids in the very same time in the very same way, some get them in the first lesson, some in the last, and some never get them.
even if its not an app, everything is so cheap and small now, 1mm camera and microphone, and very tiny ear piece (invisible with longer hair), or even earless headphones, realtime whisper turbo and groq llama3.3 can absolutely pass most interviews and answer most questions
even without the tech, i am sure people are now cheating on face to face interviews as well
i worked at fairly big company, and did maybe 500 interviews, i never knew how the candidate would look in advance, so i dont see why cant someone else shows up to the interview, pass it and then the real person shows up for the job
Their friends who have kids?
People still talk to other people.
there are more extreme cases
On his first day of work he missed the bus, so he called our boss and just said "I missed the bus, the next one is tomorrow morning, see you tomorrow".
Working on any kind of problem with him is absolute joy, from outages to designing new features, including once when the server burned down and everything was lost.
sadly after few years faster CPUs are normalized and we write sloppier code that makes the programs slow again
then amd has to work on next generation of speculative execution innovations and almost AGI branch predictors :) and we go again
or various accidentally quadratic functions
of course they shouldnt, but such is life
His companies are making rockets, autonomous humanoid robots and autonomous cars, I will cut my left pinky finger to work on any of those, and this is my capslock finger, so I will have to switch away from emacs..
With the worldlabs[1] work maybe they unlock models that do have even better spatial world model and can be used label data even better and faster, or create even better synthetic data so it can unlock FSD even sooner.
The fact that LLMs work means there is structure in language that is beyond our understanding, and yet the transformer can discover it and program itself to solve for it. I think that the stupendous amount of compute that is going to be released in 2025 will make it possible to train labelers that can do temporal labeling much better than humans and than the current models, and synthesize and perturb data to train really really good transformers that will outperform 90th percentile humans.
Why do you guys think that it can not happen? Maybe its not Tesla that does it, but I certainly think amazing tech is coming.
Hopefully it will not be just autonomous humanoid robots with guns paid by the military and patrolling the borders :astonished face:
[1]: https://www.youtube.com/watch?v=vIXfYFB7aBI (Fei-Fei Li and Justin Johnson)