GPT-Migrate converts repos from one lang/framework to another
github.com
github.com
1. Author
2. Copyright holder
3. Copyright license
...of the code generated by this tool?
Unless the answer to all of these is unambiguously "the original", then you shouldn't be using such tool on any code, especially on your employer's intelectual property.
Sorry to be so negative about it but this is something that I see skipped over in all discussions related to AI. Just because it's AI does not make it immune to copyright law. You're giving away your code to a 3rd party company under their terms and conditions, and receivig some new code back, again under their terms and conditions. The fact that it uses AI under the hood is irrelevant, you're dealing with a business that produces you an output and you should know the terms before submitting anything to them, especially if you don't own that thing.
- the copyright status of the output of LLMs is ambiguous
- this ambiguity represents a legal risk to the users of LLMs
- given that ambiguity, nobody should use LLMs for work that is intended to be copyrighted
It’s not clear to me if any of these are true individually, never mind all together.
Maybe the copyright status is ambiguous but I think the probability that the output of LLMs is owned by anyone other than API caller is very low. You can copyright works, but you can’t copyright the ideas within those works.
Possibly this represents a liability issue but I think the probability of that is vanishingly small. Just because a legal theory exists doesn’t mean it’s going to be enforceable - one of the reasons for the existence of Uber, youtube, etc. if it’s a fait-accompli it’s not a risk.
Finally, it representing a risk of copyright liability is still just a business decision. I would guess that existing software projects are riddled with code snippets of ambiguous or incompatible licenses. All it takes is one person to copy a function that is GPL and label it otherwise and then it’s out in the world, technically running amok. It is probable that under the strictest definitions, every software company is engaging in some low level of copyright infringement. All-in, this means that while it’s a risk, it’s a risk you’re probably already taking and so you might reasonably conclude to ignore.
Any lawyer will tell you if the first two points are unclear the third point is rock solid - don't use tools that have ambiguous copyright terms until AFTER the big legal fallout/legislation unless you are willing to bet the entire farm.
If the law is so unclear that lawyers can't determine legality and are waiting for additional guidance from the lawmakers, shouldn't it be legal by default?
A law like this seems pretty much impossible to enforce, though. Even if it turns out that GPT is just parroting remixes of GPL code, all examples of generated code I've seen are fully indistinguishable from code someone may have come up on their own - with the exception of contrived cases where someone is actively trying to get it to output a particular fragment of code verbatim.
My common sense says that without a tiny bit of doubt, copilot is utterly outlaw so far, essentially stealing gpl code where the only price was credit. It probably has several other problems but that is one that anyone can see. I don't mean everyone agrees, I mean the information needed to make the judgement is all present to all observers without needing to see the code or the training process or be a lawyer. There are countless examples of regurgitated gpl code, with no credit. End of search.
It's inexcusable because it would have been almost zero burden to get everyone's consent to be included in a collective credit. They didn't even do that.
So, whether or not this assertion of mine ever amounts to anything, it's a simple example of something being illegal all along, but maybe you don't know it until after you do it, and it gets examined and judged, and the judgement goes the way that burns you.
It doesn't require any new law.
Creators need to use restrictive licenses, then all of these parasitical corporations will cease to exist.
I can't talk about licensing for content creators (like youtube), because I do not have much experience about it.
I don't know which forums you are discussing these things in, but this is the first comment on all discussions about AI on HN.
I’m sure these question can be easily answered by looking it up, so why would LLMs be any different? You have control over what the LLM generates (prompts).
Consider a hypothetical LLM that was trained on data having a single undisputed copywrite owner. What would be the legal status of it's output?
In that case the tool would almost certainly generate a derivative work, which would be a copyright violation. It's the same as if I took sick strong inspiration from a certain song that I wrote a new one with the same melody and chords, which has happened a bunch of times.
But generally LLMs are most useful when they're trained on a broad enough corpus to avoid these issues.
Anyway, now consider an LLM that was trained on two corpuses(?) with two distinct undisputed owners.
Never seen it questioned that Windsor & Newton could actually be the copyright holder for half the world's art.
If I open up Photoshop and make a new file, I own my art. So why would me opening up an existing PSD and moving all the pieces around and then claiming it as my own be an issue?
I don’t think anyone would question this at all if the model were trained only on your own data. It’s the part where a bunch of other people’s stuff was involved that makes it fuzzy enough to be an open legal question.
I can sort of see that but a) I'm also seeing it posited that the copyright could belong to either ht model or the company that produced the model (where does that logic come from?) and b) My own creativity is trained on the works of many others but my work is still my own, why is this different for an LLM?
I think a same result though is that any work translated by the original author counts as a translation that maintains but doesn’t extend copyright.
The question is more like this, if google books has an api that lets you retrieve one sentence from any book, and you use it to fetch whole chapters of valuable books written by actual insightful authors.
Do you really deserve to claim to have written the book you patch together from that? Especially, do you really deserve to not only benefit from selling the new book, but not even acknowledge nor pay a percentage to the original authors?
When you write a program using a library, there is no such ambiguity or credit-washing. You wrote your app, the library authors wrote the library.
copilot is essentially stripping that. You get to not only write your app, but look like, and even, apparently, feel like, you produced everything that went into it. All that stuff that was done by someone else and gifted to you, well that's all just the tools you the real artist used. It's absurd to credit a paint brush eh?
No one was asking those questions when the backbone of any modern tech company was, ironically, thousands of copy/pasted lines of code from StackOverflow and any other references that come up in a Google search. The number of SDE’s I know who can write code without a web browser is vanishingly small.
If you disagree, how many users are on your twitter clone right now? Exactly.
Don’t trust OpenAI? Use Microsoft.
Don’t trust Microsoft? Run TII Falcon locally.
I'm not the author of this project, but in my understanding, it's the same as if you were to write the code yourself. The project doesn't publish anything and it works entirely locally aside from LLM calls (which could in the future be 100% local as well). So you remain the author and have complete control over the license of the generated code.
If I run an OCR software 100% locally, do I get the copyright on the scanned result of Harry Potter?
Don't understand: using a giant LLM locally negates all the copyrights contained in the LLM? In which country is that a law?
The license is the one of the original, since this is unambiguously a derivative work.
So not entirely locally. Yes these could eventually also run locally but OP’s point still stands.
If it runs 100% locally then yes, it would be safe to use.
openai's official stance is that it will never use API calls as training data, and that in my understanding it may retain API call data for up to 30 days for compliance purposes, but that it legally won't store it beyond that (whereas chatgpt convos are meant to be stored and used for training purposes).
as a next step, they could provide a swappable version of the LLM provider using something like https://github.com/imartinez/privateGPT, https://github.com/alexanderatallah/window.ai, etc. would love to have a standard develop here as the community matures around LLM usage
I find this reasoning a bit flimsy. Copyright doesn't have anything to do with publishing, and is it really that clear that you are the sole author of the derived work?
So swapping languages, yeah maybe, but I expect of more practical use would be the situation where you inherit a legacy codebase in an ancient version of a language or framework that hasn't been loved in a long time. I saw this so many times when doing dev team for hire work.
Obviously you'd want to do boat loads of testing and there may well be manual work left to do afterwards, but I think it would be the kind of manual work that felt like you were polishing something new and clean and beautiful rather than trying to apply bits of sticky tape to something unmaintainable.
I also wonder about eventually being able to say to an LLM "take this codebase and make it look like my code", or maybe one of your favourite open source developer's code. Maybe everyone could end up with their own code style vector attached to their github profile describing their style. You could find devs with styles close to yours to work on your team, or maybe find devs with styles different to yours so you could go and argue about tabs vs spaces or something.
I'm a bit intimidated that Josh came so close in just a week of work but it's also inspiring confidence that this is the right track and it's actually going to work when all the puzzle pieces fall into place.
edit: damn, this project actually creates rudimentary tests as well. It's such a lean approach, makes me feel like I'm still coding in 2022 when Josh is firmly in 2023.
That's exactly what LLMs don't do. In my experience, there is no way to convince ChatGPT (even v4) to follow any conventions or obey any rules. It might try a bit, but it always ends up writing everything its own way, usually as verbosely as possible.
> Given this schema ... some create statement for a table > Use this as a template ... some unrelated function > I want you to implement this ... some pseudo code
I'm exploring this problem space - this is a wide spread pain point across almost all companies that are older than 3 years old. I've seen this at tech startups as well as very large companies and anything in between. Dropbox is arguably a top tier engineering organization that probably manages tech debt as responsibly as can be expected and they still had to make major investments to move their codebase forward from various eras of web tech 1) https://dropbox.tech/frontend/the-great-coffeescript-to-type... 2) https://dropbox.tech/frontend/edison-webserver-a-faster-more... Everyone else is much worse off so the investment required to move forward is usually immense. This leads to full rewrites. Which is nice but error prone and sometimes entails huge opportunity costs
> Obviously you'd want to do boat loads of testing and there may well be manual work left to do afterwards, but I think it would be the kind of manual work that felt like you were polishing something new and clean and beautiful rather than trying to apply bits of sticky tape to something unmaintainable.
Agreed. In my opinion, now's a great time to get started with a semi automated approach like this while betting that the program synthesis and code generation capabilities will rapidly improve over the next few years. Larger context windows, solutions for hallucinations / reliability and better training data will help reduce the manual labor required.
> I also wonder about eventually being able to say to an LLM "take this codebase and make it look like my code", or maybe one of your favourite open source developer's code. Maybe everyone could end up with their own code style vector attached to their github profile describing their style. You could find devs with styles close to yours to work on your team, or maybe find devs with styles different to yours so you could go and argue about tabs vs spaces or something.
I've been thinking that personal style / training / fine tuning could become somewhat of an asset. "You are a principal software engineer at Google with particular expertise migrating codebases from {sourcelang} to {targetlang}." works fine but imagining a much richer portable input would possibly be quite valuable.
I tried to migrate a twenty year old Visual Basic 6.0 project to c# by doing it piecemeal with GPT4 and it failed completely. Both in the UI and the backend. I am keeping my fingers crossed for a GPT n+1 that actually can do this.
Incidentally, I found out that GPT4 (as in chatGPT) is very useful if you need to program in VB6 which is nearly absent from search results these days.
I tried asking ChatGPT to do it. It looked good on the surface, but it had subtly mangled some of the if conditions, etc, which resulted in it producing completely different numbers from what the original Fortran code did.
Then I remembered good old f2c, and I tried using that. Unlike ChatGPT, the code f2c produced was (as far as I can tell) correct, albeit a lot uglier. But it is a lot easier to refactor ugly-but-correct code into nicer-and-correct code, than incorrect code into correct code.
Having done a year-long stint at a mainframe team in one of the large financial corps (no, bigger than that) I can assure you that this is never going to happen for COBOL-to-java (or to-anything) unless there are strong guarantees of 100% correctness. See, one of the first things they tell you when you join a COBOL team is that you don't touch the code, unless you've filed a form that explains every detail of the change you want to make and why. In the team I worked for, that was a 10+ page Word form that would put a herd of elephants to sleep with its obstinacy and recalcitrance.
And that was only to change some JCL scripts- the scripts that run the COBOL jobs. Nobody dares to change now 50-years old COBOL code. Because every time they do, the corp loses millions. So I was told by those who knew better than me, and had been doing that job all their lives.
Bottom line, until someone figures out how to transform a gigantic, half a century-old COBOL codebase into java without breaking nothing at all, there's not gon' be any migrations.
I get a feeling that the requirements for scientific code are going to be much looser, and that this is going to cause a whole lot of mayhem, on the other hand.
Essentially, you want to build an artificial programmer ("junior dev") who can work with a better developer/manager. That seems to be the way in the short-term. Doing this by just single-shot text transformation is a lot harder. Humans can't really do that neither.
I've been translating between c# and python and having a great deal of success at the function and class level. I even ported unit tests easily between xunit and pythons unittest library. I've got close to 100% test coverage so I'm fairly confident it's done well
translating a 20yo project written in a language that isn't used much anymore is a lot different
The code looks good, and that's a great success. What else would you like?
In particular the way that object destruction works is completely different. VB6 using reference counting and .Net languages use a garbage collector.
Systems using reference counting destroy objects and run the destructors as soon as there are no references to the object while those using garbage collectors might only dispose of the objects when memory is low, perhaps never. This means that object lifetime can be very different and that patterns such as RAII require extra work in .Net.
What exactly is more modern about a system that does not have deterministic finalization as part of object scope?
The first is API hallucination, which hits as soon as you drop down into non "major" repository packages. Even GPT-4 acts like 3.5, and will cheerfully make up / use old API interfaces, pretend it knows newer versions that it does not know, and generally loop you around in very, very convincing-looking code that just does not work.
The second is style related. In particular, Go is picky with its error return semantics, and GPT-4 doesn't worry too much about this; I'm remembering a particularly subtle and annoying deadlock where it didn't defer closing a database connection inside a go routine, or alternately check for an error, and close the handle.
On balance, both of these seem super, super solvable, either by a custom LLM, or a next version with updated training. I think of GPT-4 as a reasonable mid-to-senior engineer in terms of output right now, and I think it's reasonable to start trying to port frameworks.
That said, I think I'd want it to do an excellent job at porting tests over first, and I'd inspect those heavily, and then I'd consider how to deliver a style guide for the target language in the prompts. By default, GPT-4 doesn't know exactly how you want things coded.
One last comment, Claude seems appealing to me here, with its longer context window. That said, I haven't been successful at fully using the context window -- e.g. "here's a tarball of a repo, please do x/y/z". I think word on the street is that the Claude folks use ALiBi, regardless, the 100k attention window from Claude feels more like one that can choose to alight on key areas of the input, not one that can take the entire 100k tokens into context.
This term is better than anything I was able to come up with.
The ability of LLMs to make up convincing looking bullshit is remarkable.
I'll share a funny (because it's just so dead wrong) thing I had: I was asking about a problem SnakeYAML (popular JVM YAML lib) and it suddenly started adding Jackson (JSON Object Mapper for JVM) annotations, insisting those would work. (Spoiler alert: no)
I think they give everyone access to the gpt-3.5-turbo-16k, but I have not found a way to request access for the 32k model.
There does seem to be an option through azures openai service: https://azure.microsoft.com/en-us/products/cognitive-service...
Python, JavaScript, Java, Ruby, PHP, C#, Go, Rust, C++, C++, C++, C, Swift, Objective-C, Kotlin, Scala, Perl, Perl, R, Lua, Groovy, TypeScript, TypeScript, JavaScript, Dart, Elm, Erlang, Elixir, F#, Haskell, Julia, Nim, PHP
C, C#, C++, Dart, Elixir, Elm, Erlang, F#, Go, Groovy, Haskell, Java, JavaScript, Julia, Kotlin, Lua, Nim, Objective-C, PHP, Perl, Python, R, Ruby, Rust, Scala, Swift, TypeScript
In other words, it doesn't really work.
The current wave of LLM applications still seems to me like someone just invented homeopathy and a whole bunch of people are convinced it's real and are trying to use it to create a cure for cancer. It's just people waving their hands about and intoning magick formulae, that don't work and don't produce anything useful at all.
I am curious to see where all this is going to end up. Is someone going figure out a way to make LLMs work for real-world er work? Are we all waiting patiently the next big LLM version to see if it can do the things that the current best-of-the-best can't?
- Image descriptions eg. Be My Eyes: https://www.bemyeyes.com/
- Summarisation and content distillation, question answering Etc: literally everywhere. This is now a solved problem (to the point of it being dull) thanks to LLMs.
- Customer service chat triage.
- helping students learn- eg khan academy, tutor Lily
- Helping create and debug software. If you don’t think is happening then you’re either living under a rock or just close to believe people are lying about how they’ve used it?
Calling this problem domain "solved" only undermines whatever point you're trying to make, unless you really believe that current LLMs which fabricate information, flip flop between answers with follow up questions, and even gaslight the user, can be called "solutions".
LLMs have proven themselves to be such untrustworthy "sources" of information and knowledge that I'm struggling to understand why anyone would even try to make this particular claim. It's trivial to refute by anyone who's used them and has been thoroughly refuted by StackOverflow Developer survey which reported that only 2.8% of developers "highly trust" the output of AI tools.
https://survey.stackoverflow.co/2023/#section-developer-tool...
Not sure what that survey has to do with anything.
Source?
Down below there's a comment pushing back on saying Summarization has been solved. Even he/she is saying hallucination is rare.
We still have to throw out > 50% of its output because a human can summarize the text much better.
Don't forget this is coming at the cost of jobs that the AI is replacing. I would say, for those people, it is hurting, if not destroying, their lives.
You would all do well to remember this.
Unskilled people are running out of things they can do or retrain to. Sending everyone to college has only destroyed the meaning of a college degree and created a surplus of postgrads who can't earn anywhere near their potential for lack of demand.
If we keep overfishing this lake, at some point (I am guessing in the near future) we will hit an inflection point where the number of people vastly outnumbered the availability of jobs, and social services will be strained to the point of collapse.
That is when the torches and pitchforks will come out, and these folks had better hope to whatever God they think exists that AI will save them. Because my bet is it won't.
What are YOU going to be doing to help these people? I advocate for UBI wherever feasible, I simply don't see a way to put these folks back to work without laying waste to whatever industry they decide to enter.
Not working yet. From the website:
Starting today, you can register for the waitlist in the Be My Eyes app.
>> - Customer service chat triage.
More specifically? Which company is currently using LLMs for this purpose?
>> - helping students learn- eg khan academy, tutor Lily
Helping them learn, but what? Per wikipedia:
Statements made in certain mathematics and physics videos have been questioned for their technical accuracy.[43]
Sounds like using LLMs will just generate more garbage teaching material.
>> - Helping create and debug software. If you don’t think is happening then you’re either living under a rock or just close to believe people are lying about how they’ve used it?
Oh, absolutely. People are absolutely lying to themselves. OpenAI's and DeepMind's systematic testing of their code-geneator LLMs make it very clear that those systems produce incorrect code the majority of the time. The best results reported are 28.8% for Codex (in perpetual preprint: https://arxiv.org/abs/2107.03374) and 29.6% for AlphaCode (preprint: https://arxiv.org/abs/2203.07814 Science, paywalled: https://www.science.org/doi/10.1126/science.abq1158) and the latter is with their special 10@k metric which basically means the LLM gets 10 guesses.
I have definitely observed people convince themselves online that CoPilot or ChatGPT even is helping them "improve their productivity" or some such. It is obvious that they are fooling themselves, badly. If you push them, they immediately say "oh yeah, it makes mistakes, but I can correct them" etc. So they just feel like it's useful, even when it's just making them do more work.
In fact, that's exactly like homeopathic self-delusions: people use it because it makes them feel better, not because it has any measurable benefit.
Btw:
>> ... you’re either living under a rock ...
Stop being a jerk.
That mistakes can be made doesn't mean time isn't being saved. Everybody makes mistakes as is and first try code isn't typically being pushed human or not. The presence of mistakes means nothing.
Frankly, you're the one who comes off delusional here. "Most people are telling me one thing but they can't be right because i believe so so they must be lying to themselves" isn't normal behavior, especially when your biggest argument that they must all be wrong and you right is that the machine makes mistakes. That's very weak.
Take a step back and really look at what you're saying.
So I have no doubt you are literally, right now, using some LLM to do stuff, I just have no doubt that it is not doing what you think it does.
You say you're an engineer? I know that means you write code, but the first thing that's drilled into engineers in training and in work is that you don't just make a thing and call it a day, you make sure to understand the properties of the thing you built and what it can do, and what it can't. Like you don't just put some planks on stilts and say "here's a bridge, come and drive your cars over it". You sit down and do the maths and decide what loads the bridge can take (and you optimally do this before building the bridge). So have you done anything like that? Do you have any way whatsoever to tell how often your system works and how often it shits itself?
I've long believed that interlingual glue should occur within llvm at the IR level. Reversing each layer of a compiler feels plausible if complex.
I'm left wondering if you could also use this to document or clean up machine generated code. Eg, some process generates a huge wad of bytecode, or autogenerated Java; a GPT tool cleans it up so you can actually do some things with it as a regularly skilled human.
Fast API - Express
It would be great if you could give it old , ugly code and you could get something better.
Maybe Intellj guys can use this tools to increase their productivity and we can get 100% correct tools that work with AST not with tokens, and can do advanced transformations and review that you can trust without having to double check it.
Edit: this was demo code I asked chatgpt to come up with in the first place, so the output had no problems license wise that the input didn't already have.
Then from JS to Python again.
Run the test and compare.
Once done, good job !
Could it generate tests to confirm behavior in the target and source?
(Optional) If you'd like GPT-Migrate to validate the unit tests it creates against your app before it tests the migrated app with them, please have your existing app exposed and use the --sourceport flag.
prolly going to fall on its face for something of this complexity, but "the next big thing always starts out looking like a toy"
https://papers.nips.cc/paper/2021/hash/0cd6a652ed1f7811192db...
(though worse, for complex cases, to one of the compared systems - see figure 2) (oops, fully disclosure: the system in question is my PhD work).
It’s the most popular language for heavens sakes. Developers really need to stop bringing their religion into open source.
Says the person complaining "why isn't my language supported"
The literal example shown in the README has targetlang set to nodejs. Maybe it's a bit odd to specify the runtime instead of the language, but in practice, that's maybe more useful.
js ⇒ python ⇒ js
and then compare the output JS w/ the input JS. could get wilder too like:
js ⇒ rust ⇒ typescript ⇒ java ⇒ js