20 karma · joined March 7, 2021
Is there a console version?
The segment thing and the convoluted different pointer math caused real gymnastics if you ever had data bigger than 64k... such as images.
I always thought of the segments as windows of 64k but moving between those windows, esp with the limited register set, required some real mental gymnastics.
Could you teach a 5 year old to drive a car? A 10 year old? A 12 year old? To drive a car requires being able to read, to have judgement about ice or rainy conditions, to anticipate a child running after a ball. By the time a human in in their mid teens they have acquired the base knowledge...
Small models need to have enough base knowledge to be able to be good enough -- even in a seemingly narrow regime. Where is that? Obviously they don't need all the obscure knowledge of a frontier model but there is some base level which is probably more than it would first seem.
Also, some software businesses use a ton of aggregated or hard to get data which needs to be synthesized and that doesn't go away even if the llm driven coding is cheap.