Hacker News Reader

Comment by namanyayg | Hacker News Reader

That's exactly what I want to do next.

Any tips on how should I get codebases and real world PRs? Are the ones on popular open source repos on GitHub sufficient? I worry that they don't really accurately reflect real world closed source experience because of the inherent selection bias.

Secondly, after getting all this, how do I evaluate which method gave better results? Should it be done by a human, or should I just plug an LLM to check?

View on HN

> I worry that they don't really accurately reflect real world closed source experience because of the inherent selection bias.

As opposed to what, yet another beginner React app? That’s what everyone seems to be testing with but none of the projects I’ve seen are reflective of a production codebase that’s years old and has been touched by a dozen developers.

Throw it at a complicated non-frontend mixed language repo like cxx-qt [1] or something, preferably where the training data doesn’t include the latest API.

[1] https://github.com/KDAB/cxx-qt

"preferably where the training data doesn’t include the latest API"

That is the reason LLM's in their current shape are pretty useless to me for most tasks.

They happily mix different versions of popular frameworks, so I have to do so much manual work to fix it, I rather do all by myself then.

Pure (common) math problems, or other domains where the tech did not change so much, like bash scripts or regex are where I can use them. But my actual code? Not really. The LLM would need to be trained only on the API version I use and that is not a thing yet, as far as I am aware.

Have you tried RAG on the docs?

Not yet, I can imagine it can improve things, but was too sceptical that it is worth the effort. Have you had good succes with it?

“…should it be done by a human?”

Sigh.

I'll do it personally in the beginning but was thinking about doing it on scale

> doing it on scale

Like cloud-scale, no-code scale, or NoSQL scale? You are confused, which shows that, maybe, you should not be using such tools with the experience that you don't have.

‘Like cloud-scale, no-code scale or NoSQL scale’.

That is the dumbest statement I have heard this week. You should perhaps refrain from commenting, at-least until you gain the modicum of intelligence that you currently don’t have.

https://www.kaggle.com/competitions/konwinski-prize

Given how eager Microsoft is to steal other people's code, perhaps the leaked Windows source code would be an option. Or perhaps Microsoft will let you train on their internal issue tracker.

If you could get an LLM to check, you could just spam solutions with any assortment of models and then use your checker to pick the best.

Except LLMs are usually really bad at checking.

Perhaps if they're some day augmented by formal methods, that might change.

Sorry, I was unclear- my statement was supposed to be proof that LLMs are bad at checking, because the approach I listed is well known to not work

Got it, now it makes sense :)

I may accidentally have been inspired from your message when I wrote the following piece, yesterday: https://yoric.github.io/post/formal-ai/

Very first thing you can tell us (or try if you haven't) is that if you re-prompt, does it give the same answer? Second can you get it to generate (consistently and repeatedly) the text that gp pointed out?

Don't need to switch to a different repo for quick test, just make it reproable on your current repo.

Not the exact text, but still decent quality. I'll play around with temperature and prompts a bit.

Reply on news.ycombinator.com