Hacker News Reader

Yea, this is exactly where I am at with Copilot.

So github copilot is pretty neat I entered this code:

  result_dataFrame['yearOverYearMin'] = year_dataFrame.groupby('partId')['unitCost'].min()
  result_dataFrame['yearOverYearMax'] = year_dataFrame.groupby('partId')['unitCost'].max()

Then when I put in result_dataFrame['yearOverYearChange'] Copilot gave me

  result_dataFrame['yearOverYearChange'] = result_dataFrame['yearOverYearMax'] - result_dataFrame['yearOverYearMin']

So it's a fancy context aware autocomplete. And big productivity booster on code entry.

It is also a lifesaver for complicated fiddly array index code. One of the most cognitively costly thing for me, decades into programming, is still anything related to array partitioning, or remapping indices. Copilot pretty quickly just gets what you’re trying to do and auto fills a correct line of code. It’s basically never wrong about this sort of thing. And even in the rare case where it’s not correct (usually because it’s mistaken about what I’m trying to do), it’s much faster for me to read the line and figure out what it does than it is to mentally slog through the problem and write the line from scratch.

Is there a way to fast track being able to try out Copilot? I put my email down a few days after it was announced and haven't heard anything.

Are you in an environment where the compiler / interpreter can do CSE across string-indexed dictionaries? If not, this code seems like it's gratuitously doing two extra dictionary lookups vs. an implementation using temporary variables. Now you may very well, and rightly, say "who cares" -- but to me, this is where I'd really see something like Copilot contribute. The example you gave saves typing, but leads to equal or lower quality "correct" code to what you would write anyway with a few more keystrokes -- but a system that could advise to consider the alternate version (with the human in the loop to confirm semantic equivalence for the particular dictionary in use, since the compiler / interpreter / advice tool probably cannot) would lead to not just more code per minute, but better code.

> result_dataFrame['yearOverYearMin'] = year_dataFrame.groupby('partId')['unitCost'].min()

> result_dataFrame['yearOverYearMax'] = year_dataFrame.groupby('partId')['unitCost'].max()

> And big productivity booster on code entry.

Try reducing the DRY violations and you might have to write less. Almost the entire right side of the first two lines is repeated and third line is only that long because it has to recover the results from the map instead of using local variables.

While tools like boxing gloves reduce the damage your face takes it is usually better to just try and stop hitting yourself first.

Sometimes 2 lines of code is just better than 6 and not worth the extra brain cycles.

There's DRY, then there's premature optimization.

I disagree that this is premature, but best practice. This example is actually a small lesson how this autocomplete can construct slightly less readable code. Of course not really a problem of the tool as you have to refactor anyway.

> Sometimes 2 lines of code is just better than 6 and not worth the extra brain cycles.

I am unsure about what could possibly be better about them? Twice the chance to misspell one of the string constants? Twice the time to write the code, which in this case seems to be the big blocker? Twice the code to update in case any of it has to change? Twice the complexity per line of code?

This is pretty much the case. It was a quickly needed report of data from our ERP system and dumped into an excel sheet to use. I had not used Pandas before but seemed like a good fit for me t build this out. The people who needed this info were not very sure about what they needed, so being un-DRY is better. This is only a maybe 2 page Jupyter sheet. So it's not a huge deal.

In any case, DRY is just an optimization theme. And optimizing for understanding/intent is a higher priority for me. Sometimes they intersect, sometimes not.

This. Copilot enables bad programming practices imo, especially through clunky APIs like pandas.

Yea, I could definitely improve this code. This was a quick excel report I build by extracting some data out of our ERP and building the report in Pandas. So those min and max values get surfaced in the output excel sheet.

It was a bit of an emergency and I was figuring Pandas out at the same time. The un-dry structure is handy for me because I can quickly tweak individual things since as a company we are still figuring out what is needed in the report.

I've had Copilot teach/tell me about things I didn't know I needed to be considering.

I was writing a parser for URL query strings (?these=things&key=value) and at one point Copilot suggested

  return [decodeURIComponent(key), decodeURIComponent(value)];

and until I saw that I didn't even know that percent encoded text in URLs is something I needed to be thinking about.

Absolutely cannot wait for this to come to SQL. Or is there a way already to adopt this?

I too was pretty skeptical of Copilot when I started using it last summer.

However it is shockingly good for filling out Python snippets - ie smarter autocomplete when teaching.

Popular libraries like Pandas, Beautiful Soup, Flask are perfect for this.

About 80% time it will fill out the code exactly they way I would want. About 10% time it will be something you want to correct or nudge.

Then about 10% of time it will be a howler or anti-pattern.

Then you simply explain it to students why it is not so great to say insert something at a beginning of a Python list.

Edit: Copilot is also great for filling out comments when teaching

# it can generate Captain Obvious what comments and also some nice why type comments as well

That's where it's at...today.

We're only a year into this thing existing.

I'd be VERY surprised if the data from all the interactions with it don't result in a product people care less and less about Microsoft harvesting their data from because of the value produced in exchange.

By 2025, either these models will have hit a wall on diminishing returns and it will take a complete pivot to some other approach to continue to see notable gains, or the products will continue to have improved at compounding rates and access will have become business critical in any industry with a modicum of competition.

> By 2025, either these models will have hit a wall on diminishing returns and it will take a complete pivot to some other approach to continue to see notable gains, or the products will continue to have improved at compounding rates and access will have become business critical in any industry with a modicum of competition.

Is there a single example in AI, or even technology as a whole, where simply continuing to apply one technique has led to compounding growth? Even Moore's Law, according to none other than Jim Keller[1], is more a consequence of thousands of individual innovations that are each quite distinct from others but that build on each other to create this compounding growth we see. There is no similar curve for AI.

In this case, GPT-3 (released in 2020) uses the same architecture as GPT-2 (released in 2019), expanded to have ~100x more parameters. It's not hard to see that compounding this process will rapidly hit diminishing returns quickly in terms of time, power consumption, cost of hardware, etc. Honestly, if Google, Amazon and Microsoft didn't see increased computational cost as a financial benefit for their cloud services, people might be willing to admit that GPT-3 is a diminishing return itself: for 100x parameters, is GPT-3 over 100x better than GPT-2?

It seems that the big quantum leaps in AI come from new architectures applied in just the right way. CNNs and now transformers (based on multi-head attention) are the ways we've found to scale this thing, but those seem to come around every 25 years or so. Even the classes of problems they solve seem to change discretely and then approach some asymptote.

Copilot will probably improve, but I doubt we will see much compounding. My best guess is that Copilot will steadily improve "arithmetically" as users react to its suggestions, or even that it will just change sporadically in response to this.

[0]https://youtu.be/Nb2tebYAaOA?t=1975

There has been some effort to quantify "AI scaling laws": ie how much performance increases as we scale up the resources involved. See section 1.2 of https://arxiv.org/pdf/2001.08361.pdf

My main takeaway from that paper is that a 2x increase in training cost improves performance by 5% (100x by 42%). I only skimmed the paper though.

To me this says that model scaling will not get us very much farther: we can probably do one more 100x but not two.

I talked to someone working on model scaling and they see the same numbers and draw a very different conclusion: my interpretation of their argument is that they view scaling money as easy versus finding new fundamental technical advances.

I hadn't seen that paper before, but it is excellent. Thank you for sharing!

> To me this says that model scaling will not get us very much farther: we can probably do one more 100x but not two.

I totally agree with this assessment, and note the absence of a GPT-4 release.

> my interpretation of their argument is that they view scaling money as easy versus finding new fundamental technical advances.

This paper only discussed transformers. Given the current pace of research, it's not a given that transformers won't be replaced by something else that has better scaling laws.

And indeed, this paper tells you the amount of compute and data you need for a transformer. But Moore's law isn't doing great lately. For the purposes of research,you may be able to train trillion-parameter models - but you will likely not run such a model on your phone.

In order to merit the large cost of both training and running predictions (which don't even fit the entire model in a single GPU for larger models) - models will need to become more parameter efficient than the vanilla transformer. Otherwise it's just too expensive.

My whole point is that the way in which AI progress can intersect itself can lead to compounding effects in ways technology has not.

One way to improve upon GPT-3 is almost certainly going to be adding discriminators into the mix:

"GeDi: A Powerful New Method for Controlling Language Models"

https://blog.salesforceairesearch.com/gedi/

And it may be that rather than it being a CNN discriminator such as in that case, it will be a second transformer:

"[2102.07074] TransGAN: Two Pure Transformers Can Make One Strong GAN, and That Can Scale Up"

https://arxiv.org/abs/2102.07074

My point is we may not need to reinvent some new model separate from transformers/GANs and the work so far as long as novel ways of intersecting trained models produces results that can in turn feed intersect with other models.

NVIDIA in particular has had some interesting focus on using AI to generate training data for AI.

"NVIDIA Omniverse Replicator Generates Synthetic Training Data for Robots | NVIDIA Technical Blog"

https://developer.nvidia.com/blog/generating-synthetic-datas...

I think it's very unlikely we need to wait 25 years for a new paradigm.

In fact, I'm willing to wager a bet that whatever new paradigm will arrive will itself be the output of AI simulating different models.

As someone else already mentioned, the scaling laws paint a different story empirically: we haven't hit diminishing returns at all, and there's no end in sight.

But more anecdotally, the first applied neural network paper in 1989 by LeCunn has pretty much the same format as the GPT paper: a large neural network trained on a large dataset (all relative to the era). https://karpathy.github.io/2022/03/14/lecun1989/

It really just seems that there are a certain number of flops you need before certain capabilities can emerge.

It's also really good at writing test cases. Once you've written a couple test cases manually it can take the "it should" string and auto-complete the body of the test with very high accuracy.

I largely agree with this. Occasionally it suggests a few lines that are more or less what I want, but rarely more than that.

One use case that I reach for relatively often is writing a comment to describe the one line of code I want when I can’t remember the specifics of the API I’m working with. E.g. # reset numpy random seed, and copilot pretty reliably gives me what I want.

I agree with your last point - with an added dash of being wary of getting dependent on sophisticated tools that have one vendor.

I'm not skeptical of the idea at all, however. I see it as the rough analogy of mathematicians using proof assistants. This is a coding assistant.

It's generally true that the vast, vast majority of code one writes is boilerplate or otherwise rote. The core logic and control flow that defines the meaningful bits of any complex program are nested in the middle of tons of error handling, logging, data munging, incidental book-keeping, and other very prosaic tasks.

It will not just be handy to have a friendly intelligence assist in these tasks. It doesn't require much "intelligence" (of the sort required for figuring out what the program should do in the first place and how to architect it).

Once the technology gets firmed up a bit, the productivity multiplier will be palpable and impossible to ignore. Programmers using these tools will complete tasks faster than ones that don't, and the market will shift.

Further extensions of this technology lead to automatic generation of unit tests, automatic refactoring, etc. These all exist as very specialized tools that are explicitly coded, but it's clear to see that adding a bit of ML-driven intuition to them would greatly increase their scope and effectiveness.

This could be useful for languages like Go where simple repetition and loops are preferred over clever language features like object or array destructuring that can add cognitive or performance overhead.

It’s also great at translation. Actually really great down down to interpolated strings containing variables.

Take English strings.en.json, copy to strings.th.json, open side by side, delete English text and watch copilot fill in the thai translation

Are the translations any good though?

Thank you, I was wondering what the real use was like.

I wonder if the usefulness varies with your approach to comments.

(Haven't used Copilot yet because it didn't have Emacs support.)