Hacker News Reader

I absolutely agree, but I find the situation incredibly funny. There are three characters here: me, the LLM, and the compiler. Two of them are robots, but they refuse to talk to each other - it's up to me to tell the LLM the bad news from the compiler.

that's a frontend issue though. If you use the python interpreter in ChatGPT, you can tell it to run the code until it works, at which point it'll do a couple of iterations before giving you code.

How many people are putting service credentials into a ChatGPT Python interpreter to see if their generated code "works"?

One of our very large customers was hit with ransomware recently. So…best practice is don’t do things like that.

You can just use your own.... There is still the risk GPT-4 gives you ~ "delete all the things" and you blindly run it on your infra, but.. it's better than what. you describe.

This sounds like the worst game of telephone

dont you already cut and paste your error message to google like everyone else?

I decided to cut out the middleman:

https://github.com/skorokithakis/sysaidmin/

This is really nice. Thumb up

Thank you!

Hm, that’s true. I guess someone could have hooked up pattern matching and google search results and iterated that directly into some IDE.

It will come.

In fact, GPT is really good at correcting its code given the compiler output. We just need to automate that.

I think that this is not done yet because LLMs costs are O(n+1) so you really don’t want it to be stuck in some loop.

What does O(n+1) mean?

O of infinity

Indeed. These are basically tech demos at this point. A marvel to see, and sometimes useful, but still extremely crude.

There's a lot of headroom for sophistication as a few more research insights are made and an experienced tooling team commits a few years to making rich multimodal/ensemble code assistants that can perform smart analyses, transformations, boilerplating, etc on your project instead of just adding some text to your file.

But it'll take a years of insight and labor to build that system up, adapt it to different industries/uses, and prove it out as engineering-ready.

People get caught up in the novelty of Copilot and ChatGPT and then imagine that the revolution arrives when some new paper comes out tomorrow (or that none will arrive because of today's limitations), but the far more likely reality is that the revolution paces as something more like the internet's -- real and profound, but unfolding gradually over the span of decades as people work hard to make new things with it.

Frankly, we may be barking up the wrong tree with LLMs. Sure they deliver novel and very marketable results, hence the insane funding, but I can’t help but feel it’s really just a parlor trick and there is a yet undiscovered algorithm that can actually deliver the AGI that we seek - AGI that reliable and precise like fictional AIs.

I think a lot of progress in the field of AI comes from random breakthroughs rather than evolution of existing ideas. For that reason, I think AGI will probably be another random breakthrough, and not just an LLM with more parameters. Good if you're NVidia, bad if you're OpenAI. The random breakthrough could happen at Google, or it could happen in some professor's basement. There is no way to know, and no way to throw money at the problem and guarantee a return.

> Good if you're NVidia, bad if you're OpenAI

Not necessarily good for Nvidia. If the new algorithm is branch heavy, don’t parallelize well, … etc. and operate better on CPUs rather than GPUs, it will be Intel, AMD, and ARM that enjoy the windfall.

Good point!

> bad if you're OpenAI

Just a few days ago there was a (somewhat cryptic) report about OpenAI developing a model that could do simple math. I'm certain they're pursuing quite a lot of research that's not directly related to LLMs.

I don't think we're heading towards AGI any time soon, but I can imagine a complex system in the next couple years which uses an LLM for text generation, but offloads its serious "thinking" on to predictable, specialized subsystems. It's easy to imagine a lot of possibilities for code in particular.

I look at it like finding a lost treasure or something like that. The government expedition with a 1 trillion dollar budget might find it first... or it might be in some guy's attic and his kid came over and happened to notice it. Spending infinite money on research increases your chances, but it's still a chance.

The protein folding project was basically solved due to money and ml.

Alpha fold.

The leap that area took thanks to it was a leap.

It's more than just something potentially better

It would be a very poor AGI if it didn't create a company that does AGI ASICs in the first few days. Minutes?

A cat is AGI.

(Well, not artificial, but you get my point.) It couldn’t design ASICs in minutes.

I hope that computerized AGIs also get the zoomies at 3AM.

A cat is a very narrow form of intelligence.

True. And yet so incredibly wider than any AI.

I can't use my cat to make a painting in the style of Van Gogh. Nor can I use my cat write an email to my engineering team that I'm laying of 45% of them, in corporate speak. Nor can I use my cat to refactor a class into a more functional pattern. Hell I can't even get my cat to hunt mice. He's a cute fucker though.

> there is a yet undiscovered algorithm that can actually deliver the AGI that we seek - AGI that reliable and precise like fictional AIs.

Nobody wants to hear this, but AGI could very well be the next invention after practical cold fusion and faster-than-light travel.

Yes, we're moving faster, but does anyone know how long the road is?

A space rocket is moving very quickly (50k kmph?), much faster than any other technology such as airliners, maglev trains, but the closest star is still several light years away.

> These are basically tech demos at this point.

I disagree with this conclusion. Crude as it is, Copilot _does_ make me faster in practice, even though I have to proofread its output. I think this also agrees with your second statement, that progress will come in small but useful improvements. If you treat Copilot like improved auto-completion, then it is very useful today.

Actually, while using syntax trees as the underlying model is a very interesting approach, there is still a lot of low-hanging fruit to make Copilot much more useful than it is today, just in terms of performance (waiting 1-2 seconds to see if there is a completion, and if it makes sense, interrupts my typing flow) as well as low-level features (e.g. mark and insert part of a suggestion, instead of inserting the whole and then removing the wrong parts).

I think the ultimate jump in capability would be achieved by making a dedicated language to be used by an LLM and training it in parallel on mainly that. The problem is that to make this actually useful you need to establish a new programming language. And making a useful programming language that is actually used by people is even more expensive and takes more time than training a state-of-the-art LLM...

Internet was hard limited on getting access in a physical world.

The first tech giving me internet reliable was DSL or 60kbytes/sec.

This tech could only be replaced with replacing or adding cables.

Nvidia is making a ton more petaflops per month than last year and a lot of money and people have moved/shifted in a very short period of time.

Even my company saw the writing and the only/primarily thing we talk about is ml.

I also don't have to wait for services like a bank or anyone else supporting emails or internet services.

We have everything already in place for a much faster speed than at any other tech before.

"Any sufficiently advanced technology is indistinguishable from magic."

-Arthur C. Clarke

In order to really solve this, you couldn't use an LLM, at least not one in any shape that we have now.

You'd need something that actually understands the language. What is a lifetime, what is scope, what are types, functions, variables, etc. Something that can contextually look at correct, incomplete, or broken programs and reason about what they're doing by knowing the rules that the language operates on and following them to a conclusion. It would also need to understand high-level design patterns and structure to not just know what's happening in the literal sense "this variable is being incremented" but also in a more abstract sense "this variable keeps track of references because this is mixed C/Python code that needs that to handle deallocation". Something that recognizes patterns both within and outside of the code, with appropriately fuzzy matching where needed.

And I think importantly you'd need to be able to query it at a variety of levels. What's happening on a line-by-line basis and what's happening at the high level at a given point of code. One OR the other isn't sufficient.

That is not a simple ask. We're a long way away from something that smart existing.

Absolutely it's not a simple ask. But research in program synthesis was making interesting progress before LLMs came along. I think it would be better to ask how ML can improve or speed up those efforts, rather than trying to make a general ML model "solve" such a complex problem as a special case.

A step in this direction, which I've been trying to figure out as a side project, and which I would love someone to scoop me on and build a real thing, is to stitch an ML model into into a (mini)kanren. Minikanren folk have built relational interpreters for small languages but not for a "real" industrial language (so far as I'm aware). These small relational interpreters can generate programs given a set of relational statements about them (e.g. "find f so that f(x1) == y1, f(x2) == y2, ..."). Because they're actually examining the full universe of programs in their small language, they will eventually find all programs that satisfy the constraints. But their search is in an order that's given by the structure/definition of the interpreter, and so for complex sets of requirements, finding the _first_ satisfying example can be unacceptably slow. But what if the search were based on the softmaxed outputs of an ML model, and you do a least cost search? Then (roughly) in the case that beam-search generation from the ML model would have produced a valid answer, you still find it in roughly the same time, but you _know_ it's valid. In the case where a valid answer requires that in a small number of key places, we have to take a choice that the model assigns low probability, then we'll find it but it takes longer. And in the case that all the choices needed to construct a valid answer are low-probability under the model, then we're basically in the same place that the vanilla minikanren search was.

I've thought a lot about this as well, and I'm convinced it's a really great way forward assuming the model can have some inference of the search space of the queries that it's going to run.

It's too easy to fall into infinite loops for something with only a naïve understanding of the questions.

What LLM text generation has shown is that you don't actually have to understand English to generate pretty decent English. You just have to have enough examples.

This is where the massive corpus of source code available on the Internet can help generate a "LSM" (large software model) if you can expose the tokens as the lexer understands them in the training set.

If your LSM sees a trillion examples of correct usage of lifetime and scope and types and so on, then in the same way that an LLM trained on English grammar will emit text with correct grammar as if it understands English, your LSM will generate software with correct syntax as if it understands the software. Whatever the definition of "understands" is in the context of an LLM.

But:

- natural language is flexible, computer languages are less so.

- "pretty decent English" still includes hallucinations. I've seen companies whose product demo for generating marketing copy just makes up a plausible review. Hallucinating methods, variables, other packages/modules yields broken code.

- the human thought behind natural language is not feasible to directly provide to a model. An IR corresponding to the source of the program is feasible to provide. A trace of the program executing is feasible to provide. Grounding an LLM in the rich exterior world that humans talk about is hard; grounding an LSM in the rich internal representations accessible to an IDE or a debugger is achievable.

"pretty decent english" is a pretty fuzzy bar.

Indeed, Chat GPT 4 and Copilot can generate "pretty decent code" that will look fine to the average human coder even when it's incorrect (making up methods or getting params wrong or slighly missing requirements or similar).

The level of precision required for "pretty decent non-trivial code" is much higher than prose that looks like it was written by an educated human, so I share the idea that if it was augmented - even in really stupid ways like asking the IDE if it would even compile, in the case of Copilot, before suggesting it to the user - it would work much better at a much lower effort than increasing it's understanding implicitly by orders of magnitude.

> you don't actually have to understand English to generate pretty decent English. You just have to have enough examples.

I would have thought babies have been showing this beyond a doubt since time immemorial.

No, because we can't look into their skulls, to figure out whether they 'understand', whatever that means.

right. we're already abstracting from English words and characters into tokens, piping code through half a compiler so the LSM is given the AST to train on doesn't seem all that far fetched.

I keep seeing these takes and I still think OthelloGPT disproves them.

So far it really looks like a sufficiently large LLM with a sufficiently large / high-quality dataset can learn basically anything (given a slightly generous interpretation of the word "sufficiently").

In case of code completion, I think just training on program output in addition to source code would already unlock huge capability boosts.

> You'd need something that actually understands the language. What is a lifetime, what is scope, what are types, functions, variables, etc.

Why is that more important for programming languages than for human natural languages?

Because if you use a somewhat odd phrasing, uncommon terms, weird sentence structure, etc - human beings are very good at figuring out what is meant. We can interpolate, we can extrapolate, we can estimate and understand context. Very incomplete language can still be understood with the right context.

A compiler cannot do these things. The code must follow the rules of the language perfectly, or the code won't compile. And further, it doesn't understand intention like a human can. It doesn't know that you intended to loop over all items in a collection - if you have an off-by-one error it'll happily compile that regardless.

An interface layer is 70% of the value for 10% of the work.

Good point about program traces.

Another area I wish Copilot understood: where my cursor will go next. Right now it can only feed me new lines, but I bet the underlying LLM is already powerful enough that with little fine-tuning, it could guess that after I edit a call to foobar() to add an argument the next thing I'll do is probably edit the definition of foobar.

It feels like there's a trove of UX improvements in there; we've barely scratched the surface.

IDEs already has this built in. The problem is more an effect of us deciding that source code is text instead of something integrated, like Smalltalk.

Well yeah, but IDEs have code completion built-in as well.

My point is, I think Copilot-style tools could do more that insert text. They could recommend lines of code to remove, give you a quick keyboard shortcut to "place cursor in next file / line you're likely to want to edit", etc.

IDEs can do some rigid versions of that (eg "Jump to next diagnostic", "Refactor", "Jump to implementation", etc) but Copilot could be the more general, more powerful version of these features.

I've been working on some basics in this direction... taking the expected type at the cursor and using this to inform exactly what type & function definitions to splice into the prompt: https://andrewblinn.com/papers/2023-MWPLS-Type-directed-Prom...

We're currently working on two forks off this... one, using the expected type information in conjunction with existing grammar constrained generation to enforce per-token type-correct generations, and two, using some program synthesis unevaluation techniques to also provided runtime trace data relevant to the current program hole we're trying to generate a completion for.

But yeah in general there is so much existing work on static and dynamic program analysis which can be applied here... I think a lot of the interesting challenges are going to be UI/UX ones... interactive processes to help more precisely specify intent and iteratively valid generations as more and more code is written autonomously.

That's interesting. In my experience, although synctatic errors do happen, they're not nearly as common as, say, hallucinating methods or tripping over the exact API of a library.

One thought I've had but haven't experimented with yet is that we could leverage a lot of the existing tooling that was made for humans - e.g. tab-complete providers like Jedi, which do some type inference behind the scenes. It's able to provide suggestions of valid members for a given cursor position, and so logits could be warped to prefer tokens which match the suggestions (so if the output at a given time is `math.sq`, `math.sqrt` would be much preferred over `math.square_root` which doesn't exist). You'd have to be a bit smart around this though because in since situations such as when using an identifier for the first time it's not yet in scope and you don't want the LLM to never create variables. Maybe some beam search shenanigans and heuristics could be enough to get useful output, but at that point it no longer seems like a quick thing to just try out :)

it's been done, in a limited way: https://arxiv.org/abs/2306.10763; we're working on extended this, but yeah, there are a lot of details to work out

Generating invalid code is a big hassle. I have used ChatGPT for generating R code & sometimes it refers to functions that don't exist. The standard deviation of [1,2,1] is 0.577, given by sd(c(1,2,1)). Sometimes I get stdev(c(1,2,3)) - there is no stdev in R. Why not have a slower mode where you run the code thru the R interpreter first, & only emit valid code that passes that step ?

The Python one that currently exists can use Pandas, but then you're using Python and not R. Other language support must be on their roadmap, only question is how far down the list it is.

This can be done pretty easily with their api if you are willing to spend some time on it

> these tools are engaging only at the level of text

Couldn't we say the same thing about almost all UNIX/Linux coreutils? There's no way to get a strongly-typed array/tree of folders, files, and metadata from `ls`; it has to be awk-ed and sed-ed into compliance. There's no way to get a strongly-typed dictionary of command-line options and what they do for pretty much every coreutil; you have to invoke `something --help` and then awk/sed that output again.

These coreutils are at their core, only stringly-typed, which makes life more difficult than it strictly needs to be.

Everything is a bag of bytes, or a stream to be manipulated. This philosophy simply leaked into LLMs.

I think GP's point is we could get much better results if we fed the LLMs with more than the input text.

It could still be text, or at least bytestreams: program traces, OpenTelemetry logs, objdump of the compiled binaries, LLVM IR dumps, compiler errors, syntax highlighting markers, etc.

This is, indeed, also a weakness of those tools.

At least chatgpt can already use a python interpreter and can also write unit tests for it.

Alone with copilot the amount of money flowing into this area is much more now than just a year ago.

And based on a Google research blog article, they use it internally with over 50% of suggestions being accepted.

The race is on and no one can afford not to play the game.

> These assistants should be able to make predictions about program _traces_, not just program source text.

Are you sure 'traces' is the right word? Not something more like ASTs?

Btw, the predict-next-token approach has the benefit of also being able to deal with broken code.

You might also want to compare http://www.incompleteideas.net/IncIdeas/BitterLesson.html and https://gwern.net/scaling-hypothesis with your idea of adding more domain specific knowledge.

You know I've been thinking about this for a while, I honestly don't think LLMs will ever be the right choice for a coding assistant. Where I've had a strange idea they could help, although it's completely unproven yet, is replacing what we traditionally think of as a developer and code. I'm envisioning, for simple application, a completely non-technical person have a conversation with a coding assistant and it something directly executable, think the tokens it outputs are JVM bytecode or something similar. I'm sure there's innumerable flaws with my thinking/approach but so far, it's been fun to think and code around.

Very well put. The current generation of these tools are incredibly useful but clearly leaving a huge amount on the table.

Just makes me more excited for future iterations!

Would be interesting to see them trained on actual syntax trees from text.

Then maybe have a separate model trained on going from syntax tree to source code.

> Then maybe have a separate model trained on going from syntax tree to source code.

I don't see why you need a model for this. But yes, this is a very cool idea.

Presumably so you don’t need to write a code generator for every language you support.

> then referenced a language version which does not yet exist

It must have gaming forums in its training data. Gamers know that the solution to all problems is to upgrade (game/drivers/OS) to the latest version.