If you think an LLM can just implement e.g. an alternative to the Linux kernel from scratch (or any other GPL licensed software), you might be falling for the marketing.

Have you seen LLMs build large, stable, pieces of software, that work well, with your own eyes? I have not. I have seen LLMs build small, well-understood things, and I've seen LLMs do super well with extremely well specified problems, but all the rest seems to be marketing (i.e. I couldn't observe or reproduce it).

Edit: I'm not talking about throwaway slop that just needs to run long enough to get funding.