attentionmech

why don't they just block the obs project and let users install it in unofficial manner while removing themselves as middleman? I mean, they have certain let's say guidelines but why go about enforcing them in this weird manner.

saw that video just now, thanks for this.

he has earned it haha.

Will checkout jeremy's lectures. I actually use his fastbook notebooks a lot to self-study.

Karpathy's style, for me is more like at the right abstraction to bring out curiosity in me towards the subject. After watching his lectures, i go on to more materials generally, and never really stop there.

agreed. that's err on my part to mention it like that. more evidence suggest that they were working on similar stuff but now the cat is out of the bag and open source got a win.

people already did: https://x.com/karpathy/status/1884678601704169965

This is cool, and timely (I wanted a neat repo like that).

I have also been working from last 2 weeks on a gpt implementation in C. Eventually it turned out to be really slow (without CUDA). But it taught me how much memory management and data management there is when implementing these systems. You are running like a loop billions of times so you need to preallocate the computational graph and stuff. If anyone wanna check out it's ~1500 LOC single file:

https://github.com/attentionmech/gpt.c/blob/main/gpt.c

I love this paradigm of reasoning by one model and actual work by another. This opens up avenues of specialization and then eventually smaller plays working on more niche things.

I found the following thread more insightful than my original comment (wish I could edit that one). A research explains why RL didn't work before this: https://x.com/its_dibya/status/1883595705736163727

people are doing all sort of experiments and reproducing the "emergence"(sorry it's not the right word) of backtracking; it's all so fun to watch.

Yea, they might be scaling is harder or may be more tricks up their sleeves when it comes to serving the model.

Plus, the speed at which it replies is amazing too. Claude/Chatgpt now seem like inefficient inference engines compared to it.

Do you think this feature i.e. 'finding smaller chunks easier to solve' comes out from the dataset these are trained on or is it more related to architecture components?

Most people I talked with don't grasp how big of an event this is. I consider is almost as similar to as what early version of linux did to OS ecosystem.

If you check failure section of their paper, they also tried other methods like MCTS and PRM which is what other labs have been obsessing about but couldn't move on from (that includes bigshots). Only team which I am aware which tried verifiable rewards is tulu but they didn't scaled it up and just left it there.

This sort of thing imo is similar to what openAI did with transformer architecture i.e. google invented it but couldn't scale it in the right direction and deepmind got busy with atari games. They had all the pieces still openai could do it. It seems to be it comes down to research leadership in what methods to choose to invest in. But yeah, the budgets big labs have, they can easily try 10 different techniques and brute force it all but seems like they are too opinionated in methods and less urgent on outcomes.

[paper] https://arxiv.org/pdf/2501.12948 [tulu] https://x.com/hamishivi/status/1881394117810500004

That's nice explanation. Is there any insights so far in the field about why chain of thought improves the capability of a model? Does it like provide model with more working memory or something in the context itself?

the tulu team saw it. but, yes nobody like scaled it to the extent deepseek did. I am surprised that the faang labs which have the best of the best didn't see this.

idk what i am doing but i am hooked on it. it's like as if it's directly interacting with dopamine of my brain.

It's commutative. Happiness also doesn't buy money.

A problem is only a problem if you don't want it.

The author got "enburdened with a shitload of money"

They are currency of reputation and status. If you have enough stars, you get invited to private parties with elites. (I am just joking, they are bookmarks who got famous)

wow, this RWKV thing blew my mind. Thank you for sharing this!

default to git --shallow in the cli can be one option here.

May be with these rules: - Per user account we only count one clone - We don't count anonymous clones

But I agree it's not like this is also without any issues

I think it's like a "upvote" thing which shows whether historically users have found the repo interesting. Even if you hide stars, there needs to be a way for the collective hivemind of github users to help each other with what repos are high quality or not right?

Even I am curious now. Can you share me the fork? I want to see what you added there and how it's added.

I think number of clones is a much better metric (it's like proof of work, it needs compute to clone a repo). For me starring a repo is liking bookmarking it, nothing else. They might as well just mark it as "Bookmarked" instead of "Starred".

interesting concept they are.

I don't know you all but know that I love to read your thoughts on HN. I open it like 20 times a day. I enjoy that I am here with all of you to witness our existence together. and that includes AI bots. You are also part of our shared world now. I hope we will be kind to each other as far as it matters.