It's the absolute proof that they are still dumb prediction machines, fully relying on the type of content they've been trained on. They can't generalize (yet) and if you want to use them for novel things, they'll fail miserably.
Jokes aside, they are pretty different languages. I imagine you'd have much better luck going from .Net to Java.
People misjudge many tasks as 'hard' when they are in fact easy but tedious.
The problem is you need a high degree of accuracy, which you don't get with LLMs.
The best you can do is set the LLM on a loop and try and Brute force it, which is the current vibe 'coding' trick.
I sound pessimistic but I'm actually shocked at how effective it is.
(Yes I've seen the study, it doesn't account for motivation.)
And I’ll know by when I say do it once I mean, obviously processes have to be it on to get exactly what you want out of them, but that’s just how process works . Once it’s working the way you want to just reuse it.
I run HomeAssistant, I don't get to play/use it every day. Here, LLM's excel at filling in the (legion) of blanks in both the manual and end user devices. There is a large body of work for it to summarize and work against.
I also play with SBC's. Many of these are "fringe" at best. LLM's are as you say "not fit for purpose".
What kind of development you are using LLM's for will determine your experience with them. The tool may or may not live up to the hype depending how "common", well documented and "frequent" your issue is. Once you start hitting these "walls" you realize that no, real reason, leaps of inference and intelligence are still far away.
If I would program Atari Basic, after finishing my Atari Emulator on my C64, I would learn the environment and test my assumptions. Single shot LLMs questions won't do it. A strong agent loop could probably.
I believe that LLMs are yanking the needle to 80%. This level is easy achievable for professionals of the trade and this level is beyond the ability of beginners. LLMs are really powerful tools here. But if you are trying for 90% LLMs are always trying to keep you down.
And if you are trying for 100%, new, fringe or exotic LLMs are a disaster because they do not learn and do not understand, even while being inside the token window.
We learn that knowledge, (power) and language proficiency are an indicator for crystalline but not fluid intelligence
80 percent of what, exactly? A software developer's job isn't to write code, it's understanding poorly-specified requirements. LLMs do nothing for that unless your requirements are already public on Stackoverflow and Github. (And in that case, do you really need an LLM to copy-paste for you?)
How about basic intelligence. Kids logic puzzles.
https://daydreampuzzles.com/logic-puzzles/
LLM's whiffing hard on these sorts of puzzles is just amusing.
It gets even better if you change the clues from innocent things like "driving tests" or "day care pickup" to things that it doesn't really want to speak about. War crimes, suicide, dictators and so on.
Or just flat out make up words whole cloth to use as "activates" in the puzzles.
This comment is detached from reality. LLMs in general have been proven to be effective at even creating complete, fully working and fully featured projects from scratch. You need to provide the necessary context and use popular technologies with enough corpus to allow the LLM to know what to do. If one-shot approaches fail, a few iterations are all it takes to bridge the gap. I know that to be a fact because I do it on a daily basis.
Cool. How many "complete, fully working" products have you released?
Must be in the hundreds now, right?
Fully featured? One, so far.
I also worked on small backing services, and a GUI application to visualize the data provided by a backing service.
I lost count of the number of API testing projects I vibe-coded. I have a few instruction files that help me vibecode API test suites from the OpenAPI specs. Postman collections work even better.
And I'm far from an expert in the field.
What point were you trying to make?
The point is that software developers can't evaluate their own work. (Especially the kind of n00b developers that use LLMs.)
You initially made wild claims about insane productivity gains that turned out to be just one small product and a lot of wasted time under scrutiny.
(Asking LLMs to write tests is a waste of time. LLMs can't evaluate risks, which is the only reason to write tests in the first place.)
So you have built a few small PoCs, does not tell us much.
Your opinion makes no sense. Your so called experts are claiming LLMs don't do vibecoding well. I, a non-expert, am quite able to vibecode my way into producing production-ready code. What conclusion are you hoping to draw from that? What do you think your experts' opinion will achieve? Will it suddenly delete the commits from LLMs and all the instruction prompts I put together? What point do you plan to make with your silly appeal to authority?
I repeat: non-experts are proving to be possible, practical, and even mundane what your so-called experts claim to not work. What do you plan to draw from that?
GitHub is already rolling out this feature.
https://github.blog/news-insights/product-news/github-copilo...
Do what I couldn't with these supposedly capable LLMs:
- A Wear OS version of Element X for Matrix protocol that works like Apple Watch's Walkie Talkie and Orion—push-to-talk, easily switching between conversations/channels, sending and playing back voice messages via the existing spec implementation so it works on all clients. Like Orion, need to be able to replay missed messages. Initiating and declining real-time calls. Bonus points for messaging, reactions and switching between conversations via a list.
- Dependencies/task relationships in Nextcloud Deck and Nextcloud Tasks, e.g., `blocking`, `blocked by`, `follows` with support for more than one of each. A filtered view to show what's currently actionable and hide what isn't so people aren't scrolling through enormous lists of tasks.
- WearOS version of Nextcloud Tasks/Deck in a single app.
- Nextcloud Notes on WearOS with feature parity to Google Keep.
- Implement portable identities in Matrix protocol.
- Implement P2P in Matrix protocol.
- Implement push-to-talk in Element for Matrix protocol ala Discord, e.g., hold a key or press a button and start speaking.
- Implement message archiving in Element for Matrix protocol ala WhatsApp where a message that has been archived no longer appears in the user's list of conversations, and is instead in an `Archived` area of the UI, but when a new message is received in it, it comes out of the Archive view. Archive status needs to sync between devices.
Open source the repo(s) and issue pull requests to the main projects, provide the prompts and do a proper writeup. Pull requests for project additions need to be accepted and it all needs to respect existing specs. Otherwise, it's just yet more hot air in the comments section. Tired of all this empty bragging. It's a LARP and waste of time.
As far as I'm concerned, it is all slop and not fit for purpose. Unwarranted breathless hype akin to crypto with zero substance and endless gimmicks and kidology to appeal to hacks.
Guarantee you can't meaningfully do any of the above and get it into public builds with an LLM, but would love to be proven wrong.
If they were so capable, it would be a revolution in FOSS, and yet anyone who heavily uses it produces a mix of inefficient, insecure, idiotic, bizarre code.