Show HN: Tabby – A self-hosted GitHub Copilot | Hacker News Reader

Show HN: Tabby – A self-hosted GitHub Copilot

github.com

627 pointswsxiaoys126 comments

I would like to introduce Tabby, which is a self-hosted alternative to GitHub Copilot that you can integrate into your hardware. While GitHub Copilot has made coding more efficient and less time-consuming by assisting developers with suggestions and completing code, it raises concerns around privacy and security.

Tabby is in its early stages, and we are excited to receive feedback from the community.

Its Github repository is located here: https://github.com/TabbyML/tabby.

We have also deployed the latest docker image to Huggingface for a live demo: https://huggingface.co/spaces/TabbyML/tabby.

Tabby is built on top of the popular Hugging Face Transformers / Triton FasterTransformer backend and is designed to be self-hosted, providing you with complete control over your data and privacy. In Tabby's next feature iteration, you can fine-tune the model to meet your project requirements.

Open articleView on HN

Copilot has so far been pretty useful to me as a "sometimes smarter intellisense". It'll frequently correctly guess the arguments I want for a function (and their types), and every once in a while I'll type `a.map(` and it'll auto-fill the transformation code I was planning on writing.

The simpler the task I'm trying to do, the better chance it has of being correct, but that's also the part where I feel I get the most benefit from it, because I already thoroughly understand exactly what I'm writing, why I'm writing it, and what it needs to look like, and Copilot sometimes saves me the 5-30s it takes to write it. Over a day, that adds up and I can move marginally faster.

It's definitely not a 100x improvement (or even a 10x improvement), but I'm glad to have it.

If this works as well, locally, to escape the privacy issue, I'll be thrilled. Checking it out.

Yes, I went into all of this thinking the tools need to handle complex code in order to really be useful. But that's the code I want to be most careful about, so it's the code I really want to architect myself, and understand thoroughly.

Making the simpler code much faster to develop leaves me a lot more time and focus for the complex work.

These tools really are amazing, even with their limitations.

i feel the same way, but the `a.map(` autofill is hit or miss for me. it'll interrupt my train of thought and it'll take me a minute to remember what i was trying to do, then i'll have to review the code to make sure it's right. and often there's a better way to do it, so i end up refactoring.

Yeah, I think it's pretty good at this. I've also done "write a test like the one above but for Bar instead of Foo" and it got the right answer. It would be better to not duplicate code like that in most cases, but sometimes it's OK.

I recently needed a script to grab all my github repos from the github api and produce a CSV with the columns URL, name, description, and it could not do that at all. Just hallucinated libraries and APIs that did not exist.

You should try passing docs about the API or lib you wanna use to the prompt, works pretty well

> to escape the privacy issue

Genuine question, do you not use GitHub for things other than copilot? It seems to me either the privacy issues of copilot are overblown or the privacy issues of GitHub itself are underblown, because they both end up with basically the same data.

This - if you're hosting code on GH you're covered by worst case unless you're worried they are collecting other telemetry.

No, you're not, unless you never edit any code that you don't commit. I write scratch code all the time, and I can think of at least a few times where I hard-coded things into these files that probably should never leave my work computer.

Agreed, would be good if you could whitelist only source accessible from the git repo (eg. won't work on gitignored files unless explicitly activated).

> Genuine question, do you not use GitHub for things other than copilot?

I certainly not use Github for everything.

Neither do I. For that matter, I don't use it for anything where privacy is a requirement (be it for personal or for business/work reasons).

That being said, I've way too often encountered people for whom git oddly only exists in the form of Github, to the point of those being synonymous to them. The concept of a self-hosted repo seems totally foreign to them, and the only local git trees they have is what their IDE created for them in the directory of their checked out or created source project. It's weird, but such people exist… and lots of them.

There's some people who only learned GitHub and never learned Git.

It's like learning react first ,and html+css later, it's doable but a bit convoluted

I have files open in VSCode that aren't committed to Git, and that I wouldn't be allowed to commit there even in private repos (customer data covered by GDPR).

In this context I think it's important to note the distinction between "Copilot for Individuals" and "Copilot for Business", because for twice the money you potentially get a lot more privacy:

> What data does Copilot for Individuals collect?

> [...] Depending on your preferred telemetry settings, GitHub Copilot may also collect and retain the following, collectively referred to as “code snippets”: source code that you are editing, related files and other files open in the same IDE or editor, URLs of repositories and files path.

> What data does Copilot for Business collect?

> [...] GitHub Copilot transmits snippets of your code from your IDE to GitHub to provide Suggestions to you. Code snippets data is only transmitted in real-time to return Suggestions, and is discarded once a Suggestion is returned. Copilot for Business does not retain any Code Snippets Data.

https://github.com/features/copilot

unless that's directly in the contract, I'd not trust Microsoft to discard anything that might be of value in a training dataset at some point in the future

say, if training is determined to be fair use

At the very least, it has its own privacy policy.

https://docs.github.com/en/site-policy/privacy-policies/gith...

I’m now curious if the CoPilot feedback loop is going to end up with someone getting a suggestion for an existing API key.

I would hope they’re training it to recognize and not suggest API keys. I would emphasize that as an explicitly important part of its initial definition-of-self/prompt.

This is frightening. I tested copilot when it first came out in beta, while I had client-owned docs, possibly containing API keys, open in the same IDE.

I wonder how long it will be until people's baseline assumption is that their shit is being used in this way... I genuinely don't understand how it isn't already.

My baseline assumption would be that an IDE running on my own box wouldn't be uploading my code somewhere unless it was to my own repositories. Understanding that for Copilot to work, it needs to write prompts to GPT, I'd assume it wasn't uploading anything other than the tab it was working in. Not just, like, scanning my whole drive and uploading everything. That's insane.

Maybe I rely too much on the community to red flag these things. I don't install questionable extensions. I wouldn't have tested copilot at all if it hadn't received such enthusiastic support (here, among other places). The fact that it isn't sandboxed to the document you're working on should make it an absolute malware pariah, and this is the first I'm hearing of it.

“privacy” sits in a broader sea that includes “control”, “customization” and “lock-in”. i’ve been using the GPT family of tools less than otherwise because they’re constantly changing in ways that do or will disrupt any workflow i integrate them into: new guard-rails are added such that new versions explicitly don’t support use-cases the old versions do, etc.

self-hosting solves those problems: if done right, it gives me a thing which is static until i explicitly choose to upgrade it, so i can actually integrate it into more longterm workflows. if enough others also do this, there’s some expectation that when i do upgrade it i’ll have a few options (models) if the mainline one degrades in some way from my ideal.

LLMs are one of the more difficult things to self-host here due to the hardware requirements. there’s enough interest from enthusiasts right now that i’m hopeful we’ll see ways to overcome that though: pooling resources, clever forms of caching, etc.

> pooling resources

though that too comes with its own set of privacy issues, especially when done at a larger scale with strangers on the internet as opposed to just with your personally known and trusted buddies… and rare are the people for whom the latter happen to contain enough people with strong enough hardware that that pool would be sufficient.

It's not too weird a position that some people are mistrustful of Microsoft and use other version control solutions, but want to be able to take advantage of these new classes of tools where they help.

Personally, yes, I do host with Github, bit previously that was a very well-defined boundary - only what I deliberately committed and pushed went to them. Now, it's potentially anything in my editor, so I make sure to use another program for scratch files. I probably don't have anything I'd really care about them getting their hands on, but still, it's better not to get in a bad habit and then slip up someday, pasting in private or sensitive details without thinking.

If I can eliminate that as a potential source of mistakes, so much the better.

API keys, passwords, email addresses, etc. We write those data with our editor but we don't commit them to vc and github never see them. Copilot must not see them too.

Copilot also helps a lot with verbose, RSI-inducing languages like C++. On days where my hands are cold or my hands are just not feeling as physically limber as usual, it's a big relief when it autocompletes a long series of symbols for me.

I know regular old LSP/Intellisense helps here but you are still often constrained to only autocompleting one token at a time, and have to type it at least partially in most cases.

I wonder if this will guide language design toward more verbose RSI-inducing syntax over time, sort of like how faster CPUs make software bloat more tolerable.

For example, explicitly declared type information helps the model with predictions, so a language with more of this kind of syntax will be a better fit for the model.

Gradually, will we see languages that are only meant to be used within the context of an AI assisted IDE?

>Gradually, will we see languages that are only meant to be used within the context of an AI assisted IDE?

Some languages basically require an ide to be productive. I spent the first 3 years of college using vim to code java, until a kindly professor pulled me aside and told me to get with the 21st century, lol. After I got used to it I couldn't believe I didn't switch sooner.

Some old arcane 'languages' are already things I wouldn't touch without gpt. When is the last time you had to write awk? By god I will never manually write a line of awk.

You can pry vim from my cold dead hands, unless I have to write C# or Java, and then, yeah, IDE all the way.

I was frustrated with it after a week and about quit it but am developing a more nuanced perspective.

Tldr; Can't imagine using it with a language that doesn't have great type checking to catch its BS, and they really need to tune how it interacts with traditional intelligence.

Been using it in VSCode with C# and Typescript. It gets in the way of the normal intellisense forcing you to use a shortcut to get out and back to intellisense.

For me this was really getting in the way because working one line at a time when you know what you need isn't its strong suite.

Stubbing out methods or even classes, or offering a "possibly correct" solution in areas you are fuzzy about is a strong point. Even stubbing out objects, methods, and etc using your existing code as a reference point...

But possible is the key. It's a bullshitter so it could randomly invent parameters or interfaces that are plausible, but not real. It also trips up on casing regularly.

All this to say; you gotta review everything it does and be on the lookout. Without the tooling help of a typed lang like c# or typescript this would be much harder.

I thought I was reaching a productive flow with this but then I loaded my proj in Rider, which didn't have copilot installed, and banged out some code and man it was frictionless in comparison.

Feel like copilot should be opt in via a shortcut, not opt out as it is. They really need to work out how to reduce the friction of using intellisense when it's the best option which it often is. But the copilot creators seem dead set on it constantly back-seat driving.

This is bang-on. You absolutely have to review everything it produces. Unfortunately I think a lot of junior and intermediate developers are going to let Copilot-generated errors into their codebases.

I was dealing with timezone stuff yesterday, which I have a lot of experience with, and I got Copilot to generate a bunch of code to deal with data in various timezones. I reviewed it and discovered that mixed in with all the multiplying of various numbers like 36000 and so on, it was basing Eastern time on UTC - 4, Pacific on UTC - 7, and so on. Simply by putting numbers like 4 and 7 in the code.

But of course that doesn’t take into account the difference between EST and EDT: what we call “Eastern time” is different depending on the season. Not to mention that there is a legislative push to eliminate daylight savings time, which means hardcoding these differences as integers is not a good idea.

This is a prime example of bullshit: an unambiguous solution, presented confidently, that is technically incorrect and also completely shorn of nuance.

Do you worry that if you use it too much you will no longer be able to bang out those simple tasks without it? It seems to me like it will make you weaker at some skills. Like how I take around 30 seconds to read a mechanical clockface.

Not GP, but I'm sure it will make me weaker at simple tasks, but that's also something I'm okay with personally!

I completely agree. Lately, I've been doing lots of green-field code development, and it shines in that area because there can often be lots and lots of boilerplate. I think Copilot makes me perhaps 20-30% more productive in that phase.

That's a lot less than 100x, but it's way more than nothing! I don't think I'd want to give it up.

Tried it in hobby projects: saved a lot of time in java, less in python. In my scala project i have something like an internal dsl, and it was not much of an improvement over intellisense. I just realized how much i think and overthink, while typing. When it completed the entire block, I’d have to stop and think over it regardless.

Local models are a different game though. I want not an autocomplete, i want to tell it “go do this, similarly how we did it there” and it does it. Basically, a high level dsl, that magically appears after a model looks at your codebase.

Did you python code have thorough types usage? I find that strongly-typed languages seem to do better for code suggestions based on existing code in your codebase. Now, why that is, I don't want to speculate, but for example it seems to generate better suggestions for Typescript than it does for Javascript. Python's type system is... well, _there_ and that's about all you can say for it, and I find that most Python I come across, even the professional stuff, is much less... "rigid" than code in typed languages. I wonder if that makes a difference in terms of usefulness for suggestions.

In python i rarely use typing consistently, mostly when pycharm gets confused and does not give me good autocomplete suggestions.

Having been using Copilot for a few weeks I would substantially agree with this.

However, just yesterday in a module with quite a lot of context and types it autocompleted a function I was pondering for converting between different schema languages.

It was subtly wrong but the basic structure was correct. I had wanted to implement this function for a while but, having not really thought it through, imagined it would be much more complicated and would have required reading the ABI spec for the target. I never would have attempted it unless it had shown me the way and that it was relatively simple.

This is probably a piece of code that saves significant manual code. Copilot doesn't need to nudge me towards writing that many high leverage pieces of code to start becoming a big multiplier.

I was working on a personal project last night, and I wanted to extract out a title between title=" and S0, I asked bing and it instantly gave me back a regex with the proper look ahead and look behind grouping.

Now theoretically, I know about those things, but even I'll admit I would have messed up the syntax and probably spent 15m on regex 101 tweaking things. The ai assist is great if you take it as a suggestion of something you should look at, rather than blindly trust it

If not for the wonderful product itself, take my upvote for the "hire" page. It's the first time I see a company presenting a formula where you can input your data and figure out how much you would get paid. Nice!

> https://tabbyml.notion.site/Compensation-Sheet-ad61218889ab4...

Thank you, I was inspired by the brilliant ideas presented in the employer handbooks of Posthog and GitLab, as acknowledged in the career page.

>location-based pay

Good to know they put that out there so people can avoid them.

And 10 years to exercise ISOs after departure. Very nice.

How does this compare to fauxpilot - https://github.com/fauxpilot/fauxpilot? Fauxpilot also uses Triton with fastertransformers and GPT-J style models (codegen).

Linked my replies to others' questions: [1], [2]

[1] https://news.ycombinator.com/item?id=35471882

[2] https://news.ycombinator.com/item?id=35471390

I hope these can be of help and answer your questions as well!

I'm confused about the premise here. The power of self-hosting such a thing is presumably that you would be able to train it on your own company's codebase as a corpus of examples to help other people in the company know how to navigate the specifics of your codebase.

But there's nothing in the introductory materials about how to train this thing.

Many of these that I've seen are more focused on keeping company data private than on training the model(s) on company data.

Which is pretty entertaining given how many of these companies host their code on Github already.

Sometimes it's just about avoiding boilerplate. But the last sentence of the post here says that fine-tuning is coming in the next release.

Yes, how could I train it on my codebase? Does the team acknowledge this is a thing people would want to do?

Or am I misunderstanding the idea here?

I'm not sure either, but I guess one benefit of self hosting is that you skip the $10/mo fee. I don't mind paying for it, but I know I'm not everyone.

Looking at https://d1muf25xaso8hp.cloudfront.net/https%3A%2F%2Fs3.amazo... it looks like it's figures out you're using javascript and trains itself on all the libraries you install. I'm guessing there's a way to ask it to look at your specific code as well, but haven't dug into it yet.

The power of self-hosting this is for organizations that don't want to send their code to third parties. The code generation happens via in context learning using large language models, so there is no training these things. The biggest benefit I'd say is privacy.

> in context learning using large language models

In which context is it learning the language models of what?

If it's learning the language models using a different context than the context of the company I work for, then it's learning not learning anything relative tho the codebase that's important to me. So what use it is?

Generally speaking, companies have their own libraries and their own style of coding. Having a language model of how someone at facebook is coding their javascript isn't going to help me at all with generating useful completions for my FORTRAN code against some 20 years of legacy code on my company's own codebase. But training it locally on the 20 years worth of legacy code sounds useful.

That's not how these large language models work. They don't need to be trained or fine tuned on the code of your company. They are so large, and their training corpus is so all encompassing that they generalise well to any codebase without any fine tuning.

That's what "in context" learning is. The input to the model will be code from your company. This input will have enough information for the model to do autocomplete/etc.

I think that this is a really cool project.

Personally, I am working at a financial institution which is regulated, and the legal team + cyber are still evaluating if there would be any problems that could arise with CoPilot.

The way the Europe is heading towards AI regulation, Italy has already banned chatGPT. It seems likely that there would be a threat to Copilot as well in the Europe.

So these kind of solutions would make a ton of sense wrt organisations adopting this.

However as a developer who has used copilot before, here are my 2 cents:

The copilot makes a lot of sense, wrt auto completion and code generation, because it understands the current context of code.

It would be a friction for the developers if we are expecting them to use the user interface this project comes with.

It would be great for this project to go in the direction of the following:

- Ability for the developers to train the model with our custom projects, so it can give suggestions wrt to our style of coding

- Extensions for popular Editors like VIM, VS Code, IntelliJ etc

Happy to share more further info if needed :)

Out of curiosity, how do companies offering self-hosted/on-prem solutions monetize their offerings?

Do they rely on legal contracts to prevent customers from using the software for free or modifying it for their own purposes?

VC money, mostly

Presumably the same way as every software company before SaaS took over a few years ago.

Most of them do it by charging money for the product, just like every other software company in the last four decades.

If we are talking about open source self-hosted specifically, it's mostly consulting, paid support, and offering managed hosting.

Wait a minute... You mean to seriously say people used to pay a one time fee for software? Next you're going to tell me in this magical Utopia companies allowed you to continue using the same version of their software even after they made updates no one asked for

A lot of them use a phone home/license key model... so it will stop working if you don't pay for the license. Where they don't, ya it's basically a trust system and most big companies would rather just pay the license fee than expose the legal risk of using software illegally.

Is this serious? There's very little to go on in the repo to establish how well thought through this is.

I don't want to mark them down for poor language skills but the style of the comments on the TabbyML GitHub profile suggests a rather casual approach, and when combined with a lack of any serious documentation or even basic details beyond a sketched architecture diagram, I kind of wonder... Is there any particular context others can point to that I may be overlooking?

And to clarify, I'm not trying to be down on the authors of the repo, it's more a surprise this has got so high on HN without much to support it

the interest is in clients side LLMs and similar technologies

this is enthusiast level territory that doesn't require professional window dressing at the moment

if you’re the kind of person that needs that then you aren’t the target audience yet

I don't need marketing and window dressing, I'm interested in evidence of it genuinely working but there's very little here.

The idea of client side LLMs is kind of obviously a winner, who wouldn't want that and who hasn't thought of that, so this shouldn't shoot to the top on that basis.

The repo looks close to thrown together vapourware. I'm keen not to undermine their efforts since we all aspire to create great things but the fact this has risen so high with so little supporting evidence says more about the current mood here.

its useful client side LLMs that people can potentially package as a commercial offering, and this is an ongoing gap in the market

So all sorts of vaporware will fill that, and people will commend other people that seemingly take the initiative instead of just thinking about it

I'm not surprised this got attention. As you say, everyone though of it, but that also means many many people are waiting for it.

There is a huge group of people who are interested in using Copilot, but are not allowed (or don't want) to share their code with a 3rd party service.

AFAIK there is Fauxpilot and now there is this second project. Of course it got attention based on the promise/idea alone, because many people are curious whether it could be useful for them.

Getting it running was as easy as `git clone && docker run` and it actually worked, which I appreciate given how involved other ML projects tend to be.

Were you able to use it in VS Code as an assistant? As far as I see, it just offers a stand-alone HTML page for coding in.

It's open source, you'll have to write your own plugin for that.

if you're interested in VS Code plugin you may wanna try Refact.ai (which also has a self-hosted option)

Yes they have a VS Code extension even though it's not mentioned in their Github readme for some reason (just search for Tabby in VS code extensions).

I tried it and it worked OK.

Just in case you didn't know that a project called Tabby exists (it was Terminus). It's a terminal (another one you could say). It's not my project, I'm just a user.

https://tabby.sh/

This looks exactly like what I need for a project I've been working on. How do you get your own code in to the model? Or is that the future fine-tuning step you're talking about?

Tabby collected the source code from the related repository and built them into a dataset. I already have some proof-of-concept pipelines built [1] and [2], but I still need some time to polish the data pipeline..

[1]: https://github.com/TabbyML/tabby/blob/main/tabby/tools/repos...

[2]: https://github.com/TabbyML/tabby/blob/main/tabby/tools/train...

Ok, looks like that makes sense! Is there a prompt token limit like OpenAI models? Codex I believe has a 8k prompt/response limit.

You can train this on all the code in your own repositories? I would assume that makes the completions a lot better?

Anyone know of a quick workaround for this?

    % docker run \
      -it --rm \
      -v ./data:/data \
      -v ./data/hf_cache:/home/app/.cache/huggingface \
      -p 5000:5000 \
      -e MODEL_NAME=TabbyML/J-350M \
      tabbyml/tabby
    Unable to find image 'tabbyml/tabby:latest' locally
    latest: Pulling from tabbyml/tabby
    docker: no matching manifest for linux/arm64/v8 in the manifest list entries.
    See 'docker run --help'.

I have an M2 Mac. I believe Docker is capable of running images compiled for different architectures using QEMU style workarounds, but is that something I can do with a one-liner or would I need to build a new image from scratch?

Previous experiments with Docker and QEMU: https://til.simonwillison.net/docker/emulate-s390x-with-qemu

On my M1, --platform linux/amd64 seems to work (I think I needed to turn rosetta on for this for some other stuff, I don't recall.)

(that said, I am currently downloading this image, and I've had problems running x86-64 stuff in docker sometimes)

It doesn't seem to work for me on an M1, even with that flag

> OSError: [Errno 38] Function not implemented

Do any of the authors of these models disclose a clear list of codebases/discussion sites these models are trained on? Are they at the least avoiding stuff clearly GNU (or such) licensed? Is the hope that there is just no way to figure it out?

Trying the demo I got "value is not a valid enumeration member; permitted: 'unknown', 'python', 'javascript'". Trying some clearly identifiable Rust code with language set to 'unknown' I got a completion in Java back. The completion made sense, was properly indented and syntactically correct, it just was the wrong programming language.

Is this a limitation of the hosted demo or the chosen model, or do I simply have to wait a bit until my favorite niche language is supported?

What sort of resources do we need to run this, particularly VRAM? Also, how does this compare to Fauxpilot?

Tabby's philosophy is to achieve a completion rate comparable to Codex/Copilot by using a model size of less than 1B, with support for BF16/FP16 that reduces VRAM requirements to 2G or less. This may seem impossible, given that the model size is 10 times smaller than that of Codex, but it is definitely achievable, especially in an on-premises environment where customers want to keep the code behind a firewall.

Related research works include [1]. (Hints: combine code search and LLM).

[1]: RepoCoder: Repository-Level Code Completion Through Iterative Retrieval and Generation https://arxiv.org/abs/2303.12570

This also reveals Tabby's roadmap beyond other OSS work like Fauxipilot :)

> philosophy is to achieve

That doesn't answer the question, can anyone without more VRAM than sense actually run it as-is or should we wait until they reach their allegedly impossible aspirational goal?

The very first line of this sort of post should be the specs required and if the trained model weights are actually available, otherwise it's just straight up clickbait.

The post says it uses "2GB Or less" of VRAM.

A 1B parameter transformer model is on the low/tiny-end of model size these days.

Models are usually trained in fp16, meaning 16 bits = 4 bytes per parameter. So a 1B model would take 4GB to begin with, and can be optimised down from there with sparsification and/or quantization. A 50% reduction in size might be noticeably worse than the original, but still useful for boilerplate or other highly predictable patterns.

Great to have a self-hosted solution, both for data privacy and quality improvement potentials.

What LLM is this using? Or did you train your own?

Tabby is tested against various open-source models based on GPT-J/GPT-NeoX, including https://huggingface.co/CarperAI/FIM-NeoX-1.3B and https://huggingface.co/Salesforce/codegen-350M-multi.

In the meantime, I'm training language-specific base models using NeoX blocks. I hope to release them soon.

The Huggingface Space demo is running on a derived model of Salesforce/codegen-350M-multi

After using Github Copilot for a couple of weeks, it doesn't seem to do much other than Stackoverflow/blog copy paste. Does anyone get much else from it when writing non-boilerplate, non-tutorial, and non-stackoverflow types of code?

I actually still prefer SO as I get to see different solutions to the same problem and related discussion. Co-pilot is great for boilerplate stuff though, which is the majority of lines for most projects these days I'd guess.

If you like seeing options, you might be interested in having the Copilot pane open while you code if you are using it. It shows usually about 4 solutions in the panel that it is thinking about for autocomplete, and you can click which one you want.

Not necessarily better than SO but I find it nice to see options.

I haven't used Copilot recently, but I have been using Codium[0] which is similar. I find it often does a reasonably good job at completing similar structures in my own projects where I can't imagine it has seen similar examples. It doesn't always do a great job, but it does save me time.

[0] https://codeium.com/

I migrated from copilot to codeium as I found it much faster and just as smart. Not sure if the speedup comes from network latency (I'm in New Zealand) or actual model calculations. That was pre GPT4 though, and I'm willing to try copilot again soon.

My favorite usage of it is when writing database migrations. I'll write my `up` case and then it will write the `down` case for me. For migrations adding multiple columns, indexes, triggers, etc. it's good about getting the right order for the down case and generating the right commands.

Is it writing meaningful code for me? Nah. Is it helpful? Certainly.

What programming language are you using? It’s better in more popular languages.

C#, #5 on the TIOBE index

I love C# and I used it extensively for a long time, but I think Copilot shines with JS/TS and Python because of the massive amount of code published in open source projects.

Not that I find it as useful as most people do, but there is a difference...

It's also not that great with go, but surprisingly (to me) still better than C#.

A bit ironic since StackOverflow began life as the C#/.NET QA site - and nevermind copilot is a Microsoft joint.

That's pretty much my experience. I'll go entire days accepting zero copilot suggestions. Another thing I dont like is if there is some bad code in any files it is parsing in my project it will suggest similar bad code.

What's wrong with using it for boilerplate ?

It costs 10$/month and probably saves me hours in stupid boilerplate, not just because it reduces typing but because the kind of typing it fixes is the boring stuff that gets me out of flow.

And boilerplate is a vague term, good example of copilot "AI" features :

  function(a, b, c, d) {
    if(a ...) { ..
    }

    if(b // starts generalizing from a case but is very context aware

    // here it's already doing c and d  
  }

and we can be talking about very non-trivial permutations here where it's obvious from context what you want to do but generalizing to cover all cases would be more work than just writing it.

And tests - sometimes it can generate a correct test with mocks just from function name, sometimes it's garbage - but it's really easy to glance at which it is.

My current experience with copilot for ~2 months is that it's really good at such boilerplate, it's annoying when it's late or fails at what I expect it but, and it makes generating boilerplate code very cheap so it can lead to a lot of duplication.

If you're good enough you know to pick up when you should apply DRY and when not, you'll get a feel when copilot is useful and it will be a decent productivity boost IMO. If they made it faster and more reliable (just for the cases I already use it for) I would pay >100$ month out of pocket for sure.

I suspect you're trying to feed to too large/vague of prompts.

I've found it amazing when working in semi-familiar programming languages. I'm primarily a Ruby dev, but currently working in Python. The languages are close enough that it's an easy transition, but they do a lot of fundamentals differently.

Previously, it'd be extremely disruptive to my thought-process to have to go lookup basic functions, like checking array lengths or running a regex on a string.

Now, I just write a comment like `# check array length` or `# on x_var, run this regex "\s+"`. Copilot spits back what I need (or at least close enough to avoid having to break context).

Even in core languages, I'm finding it very useful for writing system calls or niche functionality that I don't use frequently. My mental model knows what needs to be done, but I don't remember the exact calls off the top of my head.

I haven't used Copilot. I find ChatGPT a lot more useful than SO because I'm less likely to bump in to "duplicate" questions (that are rarely duplicates) and where the only interactions are criticism over the question's format or structure. I assume using Copilot lets you avoid similar problems.

I've been using it to help write python for some scientific calcs which I know that nobody else should be using on stack or github. Certainly not in the way I'm implementing them.

Copilot is fantastic at predicting where I'm heading next once I give it a little bit of a start. It helps me work out unfamiliar functions that I might need to chain, or syntax I'm not entirely familiar with.

I'd say it's a big winner for me.

Considering it's an alpha version I think a VSCode extension is the missing part

There it is: https://marketplace.visualstudio.com/items?itemName=TabbyML....

It's kinda funny that the same people knocking GPT-3 copilot are going to be losing their absolute minds when copilot X drops.

GPT-4 can autogenerate most code a business needs, just need a lone engineer to keep it in check.

A lone "senior" engineer could, maybe, but definitely not someone who hasn't worked in the problem space

Uhh okay that's every dev team downsized to one person.

Does this come pre-trained, or does it require training? What sources of data are you training on if so?

I restrict all usage of AI tools trained from publicly-sourced data because of an unknown copyright restriction, general unease, and lawsuits; however if this can be trained solely on my own codebases that are of clean providence, I can be 100% guaranteed against potential lawsuits.

Copilot is a cool tool, but super scary from a legal perspective. And even more heavily regulated industries (that I'm not in) would absolutely need their own firewalled version.

I'm glad to see projects like this.

I had a look at the demo (https://huggingface.co/spaces/TabbyML/tabby) and wasn't too impressed with the generated code for the default sample prompt (binary search) -- it recurses infinitely if the item is missing. It would be interesting to compare with Copilot's output. No idea how one would go about fixing this (other than manually add a correct binary search implementation to the training data, which feels like cheating).

Request to https://tabbyml-tabby.hf.space/v1/completions:

    {
      "language": "python",
      "prompt": "def binarySearch(arr, left, right, x):\n    mid = (left +"
    }

Response:

    {
      "id": "cmpl-...",
      "created": 1680867355,
      "choices": [
        {
          "index": 0,
          "text": " right) >> 1\n    if x < arr[mid]:\n        return binarySearch(arr, left, mid - 1, x)\n    elif x > arr[mid]:\n        return binarySearch(arr, mid + 1, right, x)\n    else:\n        return mid"
        }
      ]
    }

Formatted code:

    def binarySearch(arr, left, right, x):
        mid = (left + right) >> 1
        if x < arr[mid]:
            return binarySearch(arr, left, mid - 1, x)
        elif x > arr[mid]:
            return binarySearch(arr, mid + 1, right, x)
        else:
            return mid

Manually written test cases:

   arr = [1, 3, 5, 7]
   print(binarySearch(arr, 0, len(arr), 5)) # 2 (correct)
   print(binarySearch(arr, 0, len(arr), 4)) # RecursionError

Runnable demo:

https://tio.run/##lY/BCoJAEIbvPsUPXVw0yCyCKG@9QB3Fg@maC7bKuI...

This is awesome, and glad to see that SkyPilot is useful in distributing Tabby on any cloud! https://github.com/TabbyML/tabby/blob/main/deployment/skypil...

I like to use my brain as my copilot

Anyone dare to guess how fast would GH copilot be if it ran locally ?

My main problem with copilot is latency/speed - I would shell out for 4090 if it meant I could use local copilot model that's super fast/low latency/explores deep suggestions.

One of the weaknesses of transformers is that inference is inherently serial, one token at a time, and the entire model needs to be read once per token. This means that inference (after prompt processing) is bounded by memory bandwidth instead of compute.

That said, local solutions always tend to be lower latency than cloud ones just because you get to skip the network.

Very interesting to have a self-hosted version of Copilot. Will definitely try this out.

I hope someone comes with the AI-powered refactoring. I think it has more practical applications than Copilot, at least from my experience in C and C++ test runs of it.

Hi, will this work on AMD Gpu? I have plenty of VRAM available

Is AMD gpu support on the roadmap?

I'm interested in this stuff but I won't be buying Nvidia cards.

My assumption is that this would not be fast enough for practical use on M1/M2 Macbooks. Is that correct?

This exakt name is already used by a big open-source projekt, https://github.com/eugeny/tabby (50k stars) maybe consider changing the name for better SEO

Thanks, I'm just so into the puns: Tab-by-ML / TabbyML / Tabby

love it

Reply on news.ycombinator.com