AI programming tools should be added to the Joel Test | Hacker News Reader

AI programming tools should be added to the Joel Test

blog.waleson.com

21 pointsjtwaleson55 comments

> There's plenty of things wrong with these tools: they are often wrong, are slow and the GPT4 ones are really expensive.

So...why should they be included?

I really worry about this "often wrong" part - you only know they are wrong if you already know what you're doing. Otherwise you end up trying to use hallucinated APIs & libraries, or produce code not better than copy & pasting StackOverflow answers (which is what the AI was trained on anyway).

If an LLM hallucinates a method name you'll find out the moment you try and run the code.

Code has a built-in form of easy fact checking, which makes it one of the most appropriate applications for LLMs. It's much harder to spot a hallucinated fact in a paragraph of prose than it is to spot a hallucinated API method.

The skills you most need to develop in order to take advantage of LLM assisted programming are code reading, code review, manual and automated testing and being really good at thinking of edge-cases that might not be covered.

It turns out these are important skills for being a great developer already - LLMs just force the issue on them a little more.

If the LLM hallucinates a Python package, someone else might notice that and sneak in a malicious version of that hallucinated package.

Or if it hallucinates a method name, it might be in a code path that goes untested. How often are people using these tools to also write comprehensive test suites?

Like I said, you have to put the work in if you're going to use these tools effectively. I write a LOT of tests both with and without the help of LLMs. I try hard never to leave any code path from an LLM (or from code I write myself) untested - always manually and usually through automated tests too.

If you don't do that, the chances LLM-generated code will introduce weird bugs are high. But the chances you yourself (or your coworkers) will introduce weird bugs are high as well - QA, code reviews and testing are important no matter what helped write the code.

Due to the nature of my current work I haven't really used GPT for coding yet, but I keep wondering isn't it easier to write code than to read and truly understand it? So how much development time was really saved, if I still care about off-by-one errors or correct identity checks in hash maps or all those edge cases I probably should care about? Those are all things much harder to spot reading than writing the code.

So I keep wondering if we just save time by introducing more unknown bugs using GPT?

I guess this also has a lot to do with what code is written. I would be much more concerned with a system level C++ library than some JavaScript CRUD.

Your comment is a great example of how intelligent humans hallucinate!

Gpt-4 often handles errors well. The generated code is easy to review if you understand what you asked for (if it generates tests and examples too-which it can). Etc

People have different experiences with any given thing. You might find your TV remote intuitive and easy to us. Your grandparents might find it overwhelmingly difficult. Which is true?

Yeah, sorry for the snark. I was mostly referring to how the comment I replied to said they hadn’t actually used GPT for coding, but had lots of ideas about its limitations etc.

If you read closely I am not talking about the limitations of GPT, but about the limitations of the developer having to fix code of others (GPT essentially being another developer). I guess it depends on the complexity of the problem. A lot of stuff is very easy to review and for me other stuff needs more time to understand than it did to write.

> If an LLM hallucinates a method name you'll find out the moment you try and run the code.

What about when it confabulates a module name that's been squatted by a malware distributor?

One of the things you need to learn to do when working with LLMs is to avoid blindly installing packages it suggests.

Just like if someone opens a PR against your project on GitHub you should review the dependencies they are trying to bring in as well.

I have absolutely seen Copilot generate valid C# code with a nasty logic bug. You really need to carefully review every line, and at that point I would usually rather just write it myself.

I'm sure there are good use cases, but I just haven't seen anything life-changing in my experiments with various LLM products. The only thing I use day to day is IntelliCode, which saves a bit of typing here and there.

While it is true that code has built-in fact checking, i would expect AI to actually do the fact checking implicitly. They are all logic in the end.

I often see GPT/Gemini propose code solutions that refer to non-existing libraries or methods inside of these libraries. Perhaps the solution is to use specialized AI for coding is more advised.

And dont use the generic ones. However these hallucinations are still facts. The issue is that humans are also prone to hallucinations; this is perhaps the most challenging aspect to solve in AI. Everything the AI says must be vetted/fact-checked.

Back to coding with AI: i noticed AI, as described by the original author, is perhaps best integrated as an assistant. It works well when given enough context and a simple task that could otherwise take hours to complete but is now in seconds. So think of:

- given this function: …

- write test cases that covers the most common inputs.

LLM’s are still long way to go to replace programmers. But at the pace it’s going, it feels scary sometimes.

This is like not using keyboards because of typos.

If my keyboard had some noticeable % chance of typing the G key when I hit the F key, I’d trash it in a heartbeat.

Yeah, guck that!

It's more like not using semi-autonomuous driving features with the car entirely relying on your expertise to realise and correct when it's making a mistake. The main difference is risk. Your own life at stake versus bugs in some production system.

But why are you pushing errors to production? You know you're allowed to fix the LLM's code output, right?

If a robot could paint your house, but made three small errors, would you refuse to use it? Or would you just fix the three small errors by painting over them?

There's some kind of John Henry complex going on in this AI discussion.

You're working under the assumption that you will be able to find the errors. I personally found reviewing code always way harder than writing it and we already push tons of bugs to production in written+reviewed code.

… If your keyboard is _introducing_ typos, _buy a new keyboard_.

yeah but I can type 300 wpm on this one

So… maybe I just do a different sort of software engineering to most people here, but is “how fast can I physically write code” _really_ the limiting factor in many cases?

it's not, but we're torturing an analogy here.

Over reliance is a major part of the issues. LLMs in other contexts help bottom performers a lot, but top performers get worse because they are lulled into trusting the AI's judgment.

I've worked a good number of hours with Claude Opus and it has never produced non-compiling code (ChatGPT 4 does that for me), but it can create quite subtle bugs, which is missed by the "just make sure it compiles"-type comments in this thread.

> Note that this is often not the same as "#9 - Do you use the best tools money can buy?" as blocking AI tools is about data security, not money.

I do think AI fits #9. The fact that current AI tools are not meeting data security requirements are due to the market demands and maturity:

- price needs to be low to attract adopters.

- low price? These service providers will hoard data

- data needs to be collected for training

So i think long term, there will be more premium AI tools that “promise” to not collect your data. Perhaps self-hosted? Self hosting with AI is not attractive, at least not for consumers or small businesses.

Self-hosted is becoming more and more possible. For the first time in forever I think it makes sense to buy a beefy personal computer.

OpenAI have paid plans that promise not to collect your data already. I think Anthropic do as well.

People seem not to trust companies which make these promises, which is unfortunate for the industry.

I don't remember the Joel test being about "add the latest hype to your dev environment"

One quibble I have with this: the items on the Joel Test are all language-independent, even if you're using a minor language with poor presence on GitHub. But that's not the case with AI codegen. GPT-3.5 was downright terrible at F#, and though Bing+GPT-4 does seem a lot better I'm not sure it's actually good enough for professional F#. I also doubt AI codegen is useful for Scheme/Racket developers who create DSLs (and of course it's totally useless for proprietary languages). So I am not sure AI codegen fits with the other items on the test.

In general I am concerned that LLMs will discourage innovation in programming language design - why write a better Python if GPT can just automate the tedium away?

There is a code-gen feature in a product I know of. The product has a proprietary language and the user base clamors for AI generation. Part of the suggestion process in the tool is to feed generated suggestions through the parser and a basic semantic validation before putting the suggestion in front of users. There is still no guarantee of correctness, but the code will parse and run thanks to this. They have to implement some limitations, because it becomes expensive to re-generate repeatedly.

I believe the knowledge I gained was NDA, so I'm keeping it generic.

> I am concerned that LLMs will discourage innovation in programming language design - why write a better Python if GPT can just automate the tedium away?

I don't think so negatively. My bet: innovation in programming language design will emerge that will make it a lot "less necessary/helpful" to use AIs.

Just one example: Quite some programmers claim that AI take a lot of "tedium" from the programming away. But what if we could create programming languages that mostly get rid of this "tedium" (e.g. by using higher-level abstractions to abstract away the tedious, repetitive tasks)?

> blocking AI tools is about data security, not money.

Why is AI treated very different than say cloud? Most companies don't have problem with putting all data in Github or AWS or Office 365, but lot of them freaks out if any AI can access the data. I don't think OpenAI/copilot enterprise plan T&C/privacy policy is very different than Github or AWS.

I think it has less to do with the fact of putting data somewhere, but with the way data is accessed. When you store something in S3 - you can encrypt it, in addition to built-in encryption. You have control (more or less) over your data in your cloud tenants and databases. Not with AI models trained on your data. You can't even extract\remove it or see how it's used. Literally - zero traceability and transparency. This is the problem, not the fact that it's not stored on your physical hardware.

I suspect when any AI model will start using patents databases for training - it will be a watershed moment for what one can do with open data. Old regulations simply would not put up any meaningful fight against volume and quality of model hallucinations, that may become valuable and patentable inventions and improvements according to the same regulations.

People freak out when their data is used to train AI models, leaking their private data. AI data is regularly mined, whereas cloud data is usually encrypted in transit and at rest.

If companies are willing to break privacy policies for AI training, why not use cloud data for AI training. Sure office all 365 data would be great for building AI.

Most encryption is either provider managed where they can easily look into data, or the customer also shares key with them(e.g. EC2 instance getting encrypted S3 data with key in EC2). I have never seen someone using AWS treating amazon as a threat.

> Why is AI treated very different than say cloud? Most companies don't have problem with putting all data in Github or AWS or Office 365

The answer is (from my job experience): it is not reallt treated differently. There exist treaties with Microsoft/Azure for Windows, Office 365, Azure DevOps, ... But these are considered as inconvenient necessities that the company would be happy if they were not needed, thus I believe these treaties are watched like a hawk.

I'm beginning to think that it's because most people have an inaccurate model of how "AI training" works.

They think that anything you say to an LLM is instantly added to its "knowledge" of the world - it has a perfect memory and hoovers up even the slightest piece of new information that you expose to it.

That's not how these things work. But it's hard to convince people of that, especially when the training data used by the models is a closely kept secret!

One of the first things we did when GPT-4 became available was talk to our Azure rep and get access to the OpenAI models that they'd partnered with Microsoft to host in Azure. Now, we have our own private, not-datamined (so they claim, contractually) API endpoint and we use an OpenAI integration in VS Code[1] to connect to, allowing anyone in the company to use it to help them code.

I also spun up an internal chat UI[2] to replace ChatGPT so people can feel comfortable discussing proprietary data with the LLM endpoint.

The only thing that would make it more secure would be running inference engines internally, but I wouldn't have access to as good of models, and I'd need a _lot_ of hardware to match the speeds.

[1] - https://marketplace.visualstudio.com/items?itemName=AndrewBu...

[2] - https://github.com/mckaywrigley/chatbot-ui (legacy branch)

I must admit that Copilot usefulness for me is on the level of IDE autocomplete. I can write code in notepad.exe if necessary, but that's not productive and getting productivity hit is not nice for wellness.

So if company will not allow me to use Copilot, that would be a negative factor from me.

Bing with Copilot is insanely good and initially I was having a tough time getting my slower teammates to adopt it for the more tedious tasks.

I don’t really use it to do anything but tedious stuff and for searching for documentation that google will refuse to show anymore. and it provides sources so you can verify. It really does feel (to me) like the magic of google search’s majestic era, like 2010-2015. It just tends to give the correct answers at an extraordinarily high rate and can be poked and prodded in the right direction without a lot of work.

Google getting rid of verbatim searches is an all time fumble for me. Prompt engineering a Google search to get it to find what you want is harder than getting gpt4 to talk about mature content.

I'm still kinda pissed about that, personally.

I wouldn't object to it doing fuzzy "Google Knows Best" searches for normies, but I really wish verbatim was still an option.

On the other hand, I rarely use Google these days anyway.

using " " to force the query still works, last I checked

I'm not seeing correct answers at an extraordinarily high rate, myself. For engineering and physics questions I get confident answers that are off by several orders of magnitude, and the sources cited tend to be only tangentially related to the query and don't answer the question themselves.

What kinds of questions are you asking?

Like the original author of this post sorta alluded to, treating it like a very junior assistant or pair programmer. Those kinds of questions. Questions I do not have time to trawl google for that are easily available knowledge. I don't expect it to be an expert - I want it to eliminate toil. It does do that.

> Like the original author of this post sorta alluded to, treating it like a very junior assistant or pair programmer.

For me, treating someone like a junior assist means teaching the respective person a huge lot of stuff in a very short time, so that he gets useful as fast as possible. :-)

It does give an answer, yes. But those are the kinds of questions that I often get where the answers are badly wrong. If it's programming, that's probably fine, it's generally pretty easy to verify. If you're asking for a statistic that seems like it ought be easily available knowledge but which you couldn't find within 1 minute of checking Wikipedia, it will probably confidently tell you something badly wrong. Make sure you double check.

It's pretty good inside Visual Studio Code as well. Sometimes the autofill does the wrong thing, which is annoying, but not often enough for me to turn it off. The live chat feature is also pretty useful.

ChatGPT-4, by comparison, seems to have gotten noticeably worse at writing code since it was first released. I have no hard data to show that, but I have a very strong feeling that it has.

Now, they're both supposedly based on the same OpenAI stuff, but I think Microsoft must be adding some "secret sauce" to Copilot. Different training data? Different system prompt?

The problem is not writing code. The problem is writing the right code and supporting it as the product evolves. AI tools result in programmers that know less about their code and therefore less able to support and less able to contribute to the direction of the product. Notice that almost all of the Joel test is about how to support code over time, not which editor to use.

Trying Codeium was like the difference between notepad and an IDE.

Unfortunately, it overheats my laptop so I can't actually use it, and I primarily do support, I don't program enough at my job to justify paying for copilot.

If the CPU use was lower, I don't see why I'd ever go without.

I haven't used Codeium in a while, but I remember at some point in VSCodium it would also spike CPU usage until I turned off some features (was it Chat, or Search? something like that).

I'm surprised Codeium actually runs on VSCodium! I should definitely try disabling chat

in fact, functions that are disabled on VSCode (chat for example) if you turn off VSCode telemetry, work just fine on VSCodium even with all telemetry set to off. So I'd say it works better haha

I agree, but they should count negatively.

Yeah I've thought this myself for a while; GPT-4 class assistance is now a standard tool for me.

If a prospect client or company bans it, it's a hard no from me.

I understand that might be too extreme a red line for some, but for me, life's too short to wait for laggards to catch up with the inevitable.

Reply on news.ycombinator.com