I feel it gives me about a 30% lift on mechanical tasks and a 60% lift on learning / unblocking in areas of ambiguity. I use GPT for the mechanical tasks and ChatGPT for the learning tasks. An example of the learning would be “explain to me the use of Box::pin in the context of rust futures” or some such, but also sometimes some common idiom I’m brain farting on. Searching Kagi will yield the answers, just more slowly and deeply embedded in some document or stackoverflow answer vomit requiring lot of wasted effort that fully distracts me from my flow. The fact I can ask follow up questions on areas of ambiguity is useful. When it hallucinates it generally means I’m in an area that’s either undefined as of yet, or is really niche. The nice thing about programming is hallucination feedback is basically instant so I then pull out Kagi and research a bit, and maybe 90% of the time it’s just not possible.
There has been some work done on generating code in a feedback cycle to winnow out hallucinations and it seems to work fairly well [1]- I think 99% of the challenges LLM face are primarily related to a lack of constraint, optimization, agency, and solver feedback. As they get integrated into a system with the ability to inform and constrain and guide using classic AI techniques their true value will be attainable. But they’re pretty useful even today.
N.b., I’m a 32 year veteran distinguished engineer level at FAANG and adjacent firms that programs daily.