numba888

That's possible only if they use the same tokens. Which likely requires they share the same tokenizer. Not sure that's the case here, R1 was built on OpenAI closed model's output.

yep, they didn't do even this a few years back

> Two studies published in the latest issue of Science have revealed that birds, reptiles, and mammals have developed complex brain circuits independently, despite sharing a common ancestor.

They didn't show those 'complex brain circuits' were not present in common ancestor. Am I missing something and this is well known? There is also nothing about fish. I suspect such complex organisms cannot exist without "excitatory and inhibitory neurons" the article talks about.

Small models are just tools. Even many of them will make only a toolset. They don't evolve in AGI by themselves. But putting them together in a structure (brain) may result in something close. Like big smart calculator. It takes more to create a 'character' similar to, say, terminator.

o1 is more than just math solver. And you cannot possibly train that much in a small model.

However smaller specialized models looks to be the right way to handle world's complexity. Sort of mixture of experts on one level above. Orchestrating them will be another problem. Possible solution is generalists model "to rule them all".

Without other groups it's racism. That reminds me my last job. On my question why I'm getting spam from some group I wasn't interested in they responded like "because we are sending it to everyone, ha-ha". Typical wokes.

The polls show majority doesn't like Trump's idea. Almost half of US understands that. From this side Canada looks like exemplar democracy. No crime, no border, no zillions of migrants/year. Probably we don't know all the details. But if not Canada then what?!

Hmm.. agent cannot do self-supervised learning without actually doing it. The trick is to keep it in a sandbox.

> doing AI research means generating new knowledge about AI itself, and I don't see any evidence of that.

Wouldn't colorization count as research? In vision domain there are a lot of papers like this. Just arranging and rearranging known blocks and getting SOTA result on some datasets. ;)

> wonder if r1 or 03 can do it, with the right instructions of course.

for o3-mini-high it's too difficult to write even simple ebt. not many implementations on the web to learn from(?) reasoning isn't strong enough for this. besides what it produced is very inefficient.

one xor, yes, but on most systems smaller memory footprint is better. there are many more nodes than iterators. so savings can be significant.

another possible optimization is when 'key' fits in pointer or two. which is usually the case.

wonder if r1 or 03 can do it, with the right instructions of course.

Looks like author never heard about single pointer double linked list. That could save him 2 pointers per each node.

The idea: for simplicity taking list. single pointer saved in the node is xor of next and previous nodes. When moving forward we xor current ptr with prev. Moving backward xor with next to get prev. That it. How do we get prev/next for xor? The only way to get in the middle of the list is using iterator. Which can hold one extra pointer. For the first/last nodes extra pointer is NULL.

The same for EB tree, as navigation can be done only using iterator it can hold extra pointer.

These are my $0.02, you are free to implement and use this algorithm. No fees, no references required.

Sounds like we need more typists and horses. Will hordes of traffic controllers help, or there is a better way?

> gave up on Langchain during my first attempt 6 months ago

Why? If it's not a secret. I'm just looking for something, not sure actually what... :-\

> isn't this dangerous? isn't the efficiency given at the expense of safety and interpretability?

Final text is only a small part of model's thinking. It's produced from embeddings which probably have much more in them. Each next token depends not only on previous, but all the intermediate values for all tokens. We don't know them, they are actually important and represent inner 'thinking'. So, LLM is still a black box. The result is usually A because of B. Sort of explanation for A, but where B came from we can only guess.

> RL spontaneously should create new additions to language to help with

Yes, but it may take millions of years. One of the main reasons of LLMs success is their amazing trainability. For every input token it produces predictable output. I.e. loss. While most RL techniques go one by one 'state'. For not tokenized output we cannot predict what it should be. Thus it can be trained only through the next tokens. Which makes it probably unstable and expensive to train, limiting the length of 'continuous' part. But looks like it's still a good idea to have.

> hings like feeling pain and pleasure

can machine feel? without that there is no AGI according to definition above.

and the second question: are animals "GI"? they don't have language and don't solve math problems, never heard of np-complete.

it's more complicated than this. I mean what you get is defined by what you put in. At first is was random or selected internet garbage + books + docs. I.e. not designed for training. Than was tuning. Now we can use trained model to generate the data designed for training. With specific qualities, in this case reasoning. And train next model. Just intuitively it can be smaller and better at what we trained it for. I showed two options how data can be generated, there are others of course.

As for humans, assuming genetically they have the same intellectual abilities, you can see the difference in development of different groups. It's mostly defined by training the better next generation. Schools are exactly for this.

> it's always true that training on the output of another model will result in a worse model.

Not convincing.

You can imagine model doing some primitive thinking and coming to conclusion. Then you can train another model on summaries. If everything goes well it will be coming to conclusions quicker. That's at least. Or it may be able solve more complex problems with the same amount of 'thinking'. It will be self-propelled evolution.

Another option is to use one model to produce 'thinking' part from known outputs. Then train another to reproduce thinking to get the right output, unknown to it initially. Using humans to create such dataset would be slow and very expensive.

PS: if it was impossible humans would be still living on the trees.

github has a bunch of them for years, the most known from Andrej Karpathy:

https://github.com/karpathy/nanoGPT

some other have MoE implemented.

> The judges should apply the law, and the same case should ideally lead to the same verdict regardless of the judge.

hmm.. :) I like this. But the reality is very different and some factors which shouldn't matter can change the outcome dramatically. Like skin colors of defendant and judge. Pointing this out can be punished as well.

AIs are new search engines today. So, you need to decide if you want visibility or not. If yes then blocking is like hitting yourself in the balls. While legal it can be painful if you 'succeed'.

and in the next lifetimes too. humans are good at making problems. only lobotomy and AGI Gods can sort of 'solve' them.

By the time book comes out it's outdated. DeepSeek has its own cut-off date.

And here is the problem: AI needs to be trained on something. Use of AI reduces the use of online forums, some of them are actively blocking access, like reddit. So, for AI to stay relevant it has to generate the knowledge by itself. Like having full control of a computer, taking queries from human supervisor, and really trying to solve. Having this sort of AI actors in online forum will benefit everyone.

> current LLMs would likely give better verdicts

He probably meant _brainwashed_ LLMs. They can consistently produce desired results if you wash them the right way. It's more about personal opinion than computation. Actually it would be fun to manipulate verdicts with prompt injections ;)

Today it's within the reach, I think. While effective context window is small for big files it should be possible to use RAGs to get relevant pieces in smaller window. That will be mostly data structures and functions' declarations. Only what's needed. Humans work the same way on non-trivial projects. Supervision should include strategic planing and subdivision to smaller tasks. Not sure, just my feeling, we are close to have capable models with big enough context windows. May be they are already good enough, just need proper orchestration.

Hmm... I can try... OAI has something called 'projects', but local handling with API calls is probably the right way of doing it. Easier to switch providers, run and debug in place. With current prices it should be like < $10/month.

Looks outdated by now. Try R1.

Seriously, I've been using LLMs for coding for a while and can say early experience was disappointing, but they get better and better fast. The latest o1 looks a lot better than 4o. It's reasonable to expect with proper human supervision and interface they will be able to handle big files and projects in a year or two. Interesting times...

anyway it's not dramatic. vs 50 for Amazon. $147 was close to historical max for NVidia. Not fair either. last month in was less than $140 average, just estimate.

> there's no evidence yet that the deepseek architecture would even yield a substantially more performant model with more compute.

It's supposed to. There was an info that the longer length of 'thinking' makes o3 model better than o1. I.e. at least at inference compute power still matters.

Looks like panic sell off, but main question is:

do models with DeepSeek architecture still scale up?

If yes, then bigger clusters will outperform in near future. NVidia wins as tide rises all boats, and them first.

If not, then it's still possible to run several models in parallel to do the same, potentially big, job. Just like humans team. All we need is to learn how to do it efficiently. This way bigger clusters win again.