I'm curious what percentage of PRs are just the AI blindly writing code and submitting a PR without testing, and which have at least been locally tested to some degree. Any OS maintainers have any insights on this?
There’s no need to test the PR when you already asked the AI to not make any mistakes.
A) tests need to pass
B) anything you write needs tests
C) the code quality must adhere to these standards
etc.etc.... Helping the LLMs that people Vibe code with, produce better quality results.
By not having these in place, it means people who want to help out, cant. because htey dont understand whats going on.
adding stuff to these files, woudl allow developers to give guidelines / guardrails for developement using these agents.
Should the barrier of entry be someone who knows how to code? or should the barrier of entry be someone who is motivated to help with open-source software.
Probably yes? QED submitting slop PRs is not helping. If "helping" is sticking it through an LLM, the developers can do that themselves with better insight and guidance? If you must help via an LLM, donate cash for tokens.
If you can't code, and cant donate cash/machine time, help by confirming issue reproductions, design, wikis, documentation, whatever.
I imagine the problem will persist if users continue to submit PRs that pass the harness without being able to validate for themselves that it actually works.
And since the training data seems to be very lacking, no amount of markdown would fix that.
The motivation to help the OSS project should also come with the obligation to learn how the software operates, at least on a conceptual level. The desire to help does not grant people the pass to sledgehammer their way into adding in a feature.
This strikes me as the ideal LLM first contribution/PR, a file explaining the projects standards and testing and structure.
> This strikes me as the ideal LLM first contribution/PR, a file explaining the projects standards and testing and structure.
Why should the project maintainers write such a file, when the info already exists within the README? It is duplicated work at best, and a definitive sign of the incapabilities of the agent to properly parse the project's contribution guidelines.
https://github.com/RPCS3/rpcs3/blob/master/README.md#contrib...
I don’t mean to pile on, but like… are you actually helping if you don’t understand the code you’re fixing, don’t understand the problem you’re addressing, and don’t understand the potential solution you’re submitting for that unknown problem? Or are you just making a lot of distracting noise so you can pat yourself on the back?
I think people need to be a bit more self-critical about what they’re actually up to, and who is actually benefiting from it. Generally, from comments like yours, the answers seem to be “self-aggrandizement” and “no one”, but people really don’t want to think they might be the bad guys.