What have people found to be the best copilot paid subscription, at the moment?
For software development work, no other model comes close.
I'm in Ireland, so I interpreted that as "Not out yet".
What I interpret it as is "Not compliant with GDPR yet".
Since OP asked about a Copilot product, we (YC W23) actually built https://double.bot to do exactly this: high quality UX + most capable models. Although we don't have a subscription right now, it's free for the time being :)
Not that I mind a subscription, I’d pay it in a heartbeat if there were a JetBrains plugin.
JetBrains plugin is on the roadmap, soon!
For reference against the actual product called “Copilot”, I would say this is actually useful, vs Copilot which I would use words like “glorified autocomplete” and “slightly better intellisense” to describe. Only really good for saving you some rote typing periodically.
The primary limit with “aider” is the GPT4 context window, so really it’s about learning to work within that as its only learning curve.
I have been curious about sourcegraph’s “Cody” product if anyone here has tried it
The output is amazing though, so I keep using it. But I feel like I have no insight or control. Maybe I should read the docs more carefully.
re getting some keys - for most tasks I have not yet found a better tool than finding a chat interface to your liking, setting a fairly generic “you’re an expert programmer” system prompt with output instructions tuned to your needs, and manually adding relevant context (copy/paste) to your messages when relevant. I can’t wait for big content windows and RAG methods to improve enough to replace all that with one of assistants, but it’s just not there yet.
The output is more or less of similar quality as far as I can tell. However, what makes me like Sourcegraph's Cody more is the more advanced interface it gives me in VSCode. It allows me to do all the things I want (get quick answers to code-related questions, refactor code, write tests, and explain code that's confusing) all one hot key away.
I've tried using my OpenAI subscription directly to use GPT4, but copy-pasting between the browser and the lack of context (that GPT has of my code) is limiting.
I've tried Copilot, but it just produces large amounts of junk code, and it gets irritating.
JetBrain's AI Assistant is built into the IDE I use, it melds well with all the pre-existing coding assistance it offers, the conversation interface is useful and can answer questions on specific parts of code (I habitually highlight whatever I'm interested in and ask it about that), and it's good when I ask it to write tests.
Then again, I haven't given it much of a fare shake perhaps. I do find AI Assistant is a lot more compelling.
Edit: Just tried it out again, honestly pretty disappointing. It works and good they released it, Github has been ignoring the Jetbrains experience. Their implementation has bloated UI and I don't like the way they handle prewritten prompts.
I really like this implementation. https://github.com/didalgolab/chatgpt-intellij-plugin
I really like it because it makes using GPT 4 or their proprietary model much easier than copying and pasting between a browser and VS code.
It is a paid subscription, but it includes 500 messages per day, which is way better than the time window on ChatGPT for me.
In particular, I haven't found any tools that can do either of the following:
- Comprehend a large codebase. All assistants seem to have a pretty narrow scope for the context, like the current file or few of its neighbors. E.g. they can write some code that already exists somewhere as a ready-to-use utility function, if this function isn't mentioned anywhere near to the location I'm working on.
- Automate anything. IDE integration is extremely minimal in everything I've seen so far. A copilot can spew a new version of the code piece that can be placed with a click, but that's the extent of it. It otherwise can't replace a human in the loop, even if the work is simple, trivial and highly repetitive. E.g. doing a regex-powered "replace all" still cannot be replaced with a natural language query.
- Do any repetitive work without making mistakes or hallucinating something. E.g. a copilot can try to generate an unit test for me and the overall structure can be quite decent, but I cannot ever be sure it won't miss or invent something if it involves creating some complex object and checking all its fields after some transformation.
What else should an IDE integrated coding copilot be able to do?
I used cody at work, and the best feature for that I found was chore work - I could highlight a struct (for example) and give it instructions how to transform it and it did a good job there.
If you are "intentional" I usually just use ChatGPT-4 directly. I've had a super pleasant experience with Cursor, too. You can just open Cursor and say "write readme for me", it will scan your project and right a decent readme. It is also eeeeextremely goooood at "I need to know where in the codebase is doing this" and it finds. But for day to day I still prefer chatgpt 4.
So I would say:
- If you want a nice autocorrect, copilot
- If you want something that knows your project and tries to help, cursor
- If you just want something smart, chatgpt with gpt-4.
I would say copilot lets me be 15% faster daily, and gpt-4 about 30% faster (when I need it), but I need it less often - maybe twice per day.
- Doesn't close brackets / adds too many closing brackets
- Spams completions while you're writing comments
- Doesn't auto import the required libraries after accepting a suggestion
- No multi cursor mode
- Refuses to name variables
- Refuses to trigger in the middle of a line
- Lower quality code than ChatGPT (probably because they use a smaller model)
Asking because I was also a heavy Github Copilot user but after 1-2 years of constant subpar UX and seemingly no effort from the Copilot team to fix it, I just went ahead and built my own extension that fixes all of the above: https://double.bot
Also it wasn't on your list but you mentioned GPT-4 is smart. You should try the new Claude 3 Opus, we implemented it on Monday and have been getting super positive feedback from users!
What about autocompleting on sensitive files like .env or so.
I used Copilot for a while, and it's no more than very smart autofill and good at generating boilerplate.
ChatGPT actually solves (some, minor) problems. I kept using Copilot because of comfort, and it having the context. But I stopped using it 2-3 months ago, and haven't looked back.
If what you are looking for a high quality UX + access to the most capable models (today that'd be the GPT-4 killer, Claude 3 Opus), then I think you'll like what we've built.
Double ha similar features to all the other Copilots, code autocomplete, chat, etc but we've put particularly care around getting the small details right (even the smallest thing like making sure our autocomplete closes the brackets appropriately). If you're coming from Github Copilot you should check out this side-by-side comparison: https://docs.double.bot/copilot
Can double's plugin accept bring your own key like cursor.sh ?
Would love to use an extension with my favorite editor instead of adapting a whole separate editor like cursor.sh
Thanks
And we just shipped support for Claude 3 (Opus and Sonnet) as well as using any local LLM via Ollama for completions and chat.
I know I liked it better than TabNine, CodeGPT and Codeium in VS Code. I'm pretty sure I tested a few others from the top of list of extensions when you search the extension marketplace, but I forgot which ones.
Blackbox just seems more intuitive in use and had better answers. They don't have many followers on Twitter, I'm surprised more people aren't talking about it.
These copilots are way more useful when their knowledge context is cross-file; anything restricted to current active file or making me do work to select some file subset I'm gonna pass on.
That being said if you want to keep playing with Claude 3 Opus, we have it available on our free tier at https://double.bot
In my experience 90% of code is 90% the same as another piece of code in the same repo, with small differences, and copilot will make you fly writing that code.
If you can't read the output code, does it mean the rest of your codebase is similarly unreadable?
The complexity in a codebase or a system is usually from different parts integrating or an overall architecture, but that's totally different to an individual function
Even though the structure of the code in the file was ok, it called some APIs that did not exist, it created a new var `this._menu` for the dropdown that was not needed (this.menu already exists) and in the end I still had to go through gnome extensions docs to figure out how to do it right.
Overall I don't regret using it but the experience wasn't magical, as I guess we all want it.
In that case I think I was better off not relying on the tool. I do find it nice to steer me in a direction, but the things I use tend to be niche enough that I don't get the benefit many others do.
I also have a feeling you and I are using it in different capacities.
This was just me interacting through text prompts. I could imagine some kind of more integrated solution where you can provide some basic test cases, and the system would run those cases through code proposed by the LLM could go a long way towards improving this.
For now it seems mainly useful as a way of getting a quick first draft of some code which I then have to fix up and get fully working myself.
Just this week we made it available on https://double.bot (VS COde Copilot extension). Have been getting similar feedback from multiple users
I had tried a simple experiment to generate some basic Go REST service, XML parsing code, using Bard and ChatGPT and they were actually not bad. But, that was a very simple and new code.