Hacker News Reader

Comment by dragonwriter | Hacker News Reader

I mean, coding contests tend not only to be well-specified, but also extremely micro tasks. A whole lot of the work of software development is taking even reasonably well-specified business requirements and determing the appropriate structure of tasks at that level to meet them.

Coding contest problems may be a common hiring filter, but they are very much not representative of software development work (OTOH, AI coding assistance, which this, Copilot, ChatGPT, etc., illustrate) are going to render them less valid as hiring filters, both because they will make them harder to use as skill tests, and because they will further reduce the role of the micro-focus skills they center in real-world software development.

View on HN

chatGPT seems amazing at taking corrections, iterating on the requirements, fixing as it goes. So the badly-specified requests are going to be sorted out by multiple rounds of interaction.

Try to get chatgpt to solve a basic math problem using BEDMAS instead of PEMDAS, even with multiple interactions and reminders it still gets incredibly confused.

ChatGPT cannot correctly perform step by step operations within strict formal systems. On the other hand, it can provide a high level sketch of how tomathematically prove something[1].

I think this is amazing, because this is the opposite of how we normally think about computers. ChatGPT is highly intuitive instead of a formal reasoning system. No doubt some future AIs will combine this intuition with checks against formal verification engines to check correctness.

[1] Section 6 of https://yaofu.notion.site/How-does-GPT-Obtain-its-Ability-Tr...

I tried giving ChatGPT an extremely easy “warmup” interview question, and it passed it with flying colors. But then I added the catch that it has to handle UTF-8 strings and not just ASCII, and it confidently spewed out the same program, claiming that by using the c library functions isspace and ispunct, unicode is handled because they let you pass a locale. This is, of course completely wrong. I even gave an example of an input it would fail on and it shrugged and returned roughly the same function that would still fail on that input.

On the plus side, this sounds a lot like the kind of mistake a human engineer would make.

Sort of, but I wouldn’t expect a human engineer to be so confidently incorrect. In fact, it would be a big red flag. I’d much rather have a “you know, I’m not very familiar with Unicode or UTF-8, so I’d have to do some research before I could come up with an answer.”

I did similar, I asked it to compute the median on some filtered time series data, but as I added more requirements to the filtering it couldn’t seem to adapt the program.

ChatGPT is an interactive prompt meant to amuse you with blatant lies, not something meant for any sort of reasonable purpose.

Reply on news.ycombinator.com