Emacs-copilot: Large language model code completion for Emacs
github.com
github.com
I believe average stats for programmer productivity of production-quality, debugged and maybe reusable code are pretty low - around 100 LOC/day, although it's easy to hit 1000 LOC/day or more when building throwaway prototypes/etc.
The difference between productivity in terms of production quality code and hacking/prototyping is because of the quality aspect, and for most competent/decent programmers coding something themselves is going to produce better quality code, that they understand, than copying something from substack or an LLM. The amount of time it'd take to analyze the copied code for correctness, lack of vulnerabilities, or even just decent design for future maintainability (much more of a factor in terms of total lifetime software cost than writing the code in the first place) would seem to swamp any time gained in not having to write the code yourself (which is basically the easiest and least time consuming part of any non-trivial software project).
I can see the use of LLMs in some learning scenarios, or for cases when writing throwaway code where quality is unimportant, but for production code I think we're still a long way from the point where the output of an LLM is going to be developer-level and doesn't need to be scrutinized/corrected to such a degree that the speed benefit of using it is completely lost!
The Duomo in Florence took multiple generations to build. Took them forever to figure out how to build a roof for the thing. Would you want to be a builder who focuses your whole life on building a house you can't live in because it has no roof? Or would you simply be proud to be taking part in getting to help lay the foundation for something that'll one day be great?
That's my dream.
No doubt there will be much better AI-based tools in the future, but I'd argue that if you want to accelerate that future then rather than applying what's available today, it'd make more sense to help develop more capable AI that can be applied tomorrow.
But since I wired Mixtral up to Emacs a few weeks ago I discovered that LLMs are crazy good at Lisp (and protobuf and prisma and other lispy stuff). GPT-4 exhibits the same property (though I think they’ve overdone it on the self-CoT prompting and it’s getting really snippy about burning compute).
My dots are now like recursively self improving.
hear, hear! I have the exact same impression, probably since the gpt-4-turbo preview rolled out
Even if I wanted to use it for code, I just can't. And it's actually make code review more difficult when I look at PRs and the only answer I get from the authors is "well, it's what GPT said." Can't even prove that it works right by putting a test on it.
In that sense it feels like shirking responsibility - just because you used an LLM to write your code doesn't mean you don't own it. The LLM won't be able to maintain it for you, after all.
Half of the stuff GPT comes up with in the reviews I could rewrite much more simply and directly, while improving code comprehension.
In my line of work (infra / automation) I may not write any new code that's going to be added to some project for later use for days, sometimes weeks.
Most of the stuff I do is root cause analysis of various system failures which require navigating multiple machines, jump hosts, setting up tunnels and reading logs.
So, the places where the lack of completion is the most annoying are, for example, when I have to compare values in some /sys/class/pci_bus/... between two different machines: once I've figured out what file I need in one machine in its sysfs, I don't have the command to read that file on the other machine, and need to retype it entirely (or copy and paste it between terminal windows).
I don't know what this autocompletion backend is capable of. I'd probably have to do some stitching to even get Emacs to autocomplete things in the terminal instead of or in addition to the shell running in it, but, in principle, it's not impossible and could have some merit.
I wonder what you mean. The `dabbrev-expand` command (bound to `M-/` but default) will complete the characters before point based on similar strings nearby, starting with strings in the current buffer before the word to complete, and extending its search to other buffers. If you have the sysfs file path in one buffer, it will use that for completion. You may need to use line mode for for non-`M-x shell` terminals to use `dabbrev-expand`.
> In my line of work (infra / automation) I may not write any new code that's going to be added to some project for later use for days, sometimes weeks. > > Most of the stuff I do is root cause analysis of various system failures which require navigating multiple machines, jump hosts, setting up tunnels and reading logs.
This sounds like an ideal use case for literate programming. Are you using org-mode? Having an org-file with source blocks would store the path string for later completion by the methods described above (as well as document the steps leading to the root cause). You could also make an explicit abbrev for the path (local or global). The document could make a unique reference or, depending on how many and how common the paths are, you could define a set of sequences to use. For example "asdf" always expands to /sys/class/pci_bus/X and "fdsa" expands to something else.
Hope that helps or inspires you to come up with a solution that works for you!
No... not at all... Most of the "code" I write in this way is shell commands mixed with all kind of utilities present on the target systems. It's so much "unique" (in a bad way) that there's no point trying to automate it. The patterns that emerge usually don't repeat nearly often enough to merit automation.
Literate programming is the other extreme, it's like carving your code in stone. Too labor intensive to be useful in the environment where you don't even remember the code you wrote the day after and in most likelihood will never need it again.
> will complete the characters before point based on similar strings nearby
They aren't nearby. They are in a different tmux pane. Also, that specific keybinding doesn't even work in terminal buffers, I'd have to remap it to something else to access it.
The larger problem here is that in my scenario Emacs isn't the one driving the completion process (it's the shell running in the terminal), for Emacs to even know those options are available as candidates for autocompletion it needs to read the shell history of multiple open terminal buffers (and when that's inside a tmux session, that's even more hops to go to get to it).
And the problem here, again, is that setting up all these particular interactions between different completion backends would be very tedious for me, but if some automatic intelligence could do it, that'd be nice.
Tramp?
Thanks for pushing the tooling of self-hosted LLMs forward, Justine. Llamafiles specifically should become a standard.
Would there be a way of connecting to a remote LLM that's hosted on the same LAN, but not on the same machine? I don't use Apple devices, but do have a capable machine on my network for this purpose. This would also allow working from less powerful devices.
Maybe the Llamafile could expose an API? This steps into LSP territory, and while there is such a project[1], leveraging Llamafiles would be great.
The call takes a "stream" boolean: stream: It allows receiving each predicted token in real-time instead of waiting for the completion to finish. To enable this, set to true.
And the response includes: stop: Boolean for use with stream to check whether the generation has stopped (Note: This is not related to stopping words array stop from input options)
Certainly the local web interface has a stop button, and I'm pretty sure that one did work.
But maybe I'm misunderstanding the challenge here?
I use this in a private fork of Chatbot-UI, and it just works.
Edit: I mean as the default. Which requires users to do a big scary --disable-security or equally scary red button to turn off. Which is what browsers do.
And more importantly, who wants to pipe all their private stuff through their servers? Given their attitude toward other people's copyrighted works its guaranteed to ingested by their model and queried in god mode by Sam Altman himself, looking for genius algorithms or ideas for his on-the-side startups.
Let's see if 288GB multi-core M3 processors with 100GbE (on 10m copper!) happen.... but there's always https://huggingface.co/blog/lyogavin/airllm
No coders on StarTrek yoh https://youtu.be/MX95usfB2ZA
My environment is configured with ZSH 5.9. If I invoke the LLM directly as root (via SUDO,) it loads up quickly into a web server and I can interact with it via a web-browser pointed to localhost:8080.
However, when I try to run the LLM from Emacs (after loading the LISP script via M-x ev-b,) I get a "Doing vfork: Exec format error." This is when trying to follow the demo in the Readme by typing C-c C-k after I type the beginning of the isPrime function.
Any ideas as to what's going wrong?
sudo wget -O /usr/bin/ape https://cosmo.zip/pub/cosmos/bin/ape-$(uname -m).elf
sudo chmod +x /usr/bin/ape
sudo sh -c "echo ':APE:M::MZqFpD::/usr/bin/ape:' >/proc/sys/fs/binfmt_misc/register"
sudo sh -c "echo ':APE-jart:M::jartsr::/usr/bin/ape:' >/proc/sys/fs/binfmt_misc/register"
Llm uis need a lot of work to match that.
./wizardcoder-python-34b-v1.0.Q5_K_M.llamafile
If most people actually seriously scrutinized the code (which you should) it’d be apparent that the value proposition of using LLMs is not increased throughout, but better quality code.
If you just accept the output without much scrutiny, sure you’ll increase your throughput, but at the cost of quality and the mental model of the system that you would have otherwise built.
For example, one key stroke could be "complete this code", but other keystrokes could be:
- send current buffer to LLM as-is
- send region to LLM
- send region to LLM, and replace with result
I guess there are a few orthogonal features. Getting input into LLM various ways (region, buffer, file, inline prompt), and then outputting the result various ways (append at point, overwrite region, put in new buffer, etc). And then you can build on top of it various automatic system prompts like code completion, prose, etc.
gptel is designed to do this. It also tries to provide the same interface to local LLMs (via ollama, GPT4All etc) and remote services (ChatGPT, Gemini, Kagi, etc).
Great work.
https://www.youtube.com/watch?v=bsRnh_brggM
Lowest friction llm experience I've ever used... You can even use it in the M-x minibuffer prompt.
> Also worth checking out for more general use of LLMs in emacs: https://github.com/karthink/gptel
From what I see, gptel is more interested in creating the best and least intrusive interface - it doesn't concern itself too much about which model you're using. The plan is to outsource the "connection" (API, local) to the LLM to another package, eventually.
But: The mode of operation is quite different from how GitHub CoPilot works, so maybe the name is not very well chosen.
It's somewhat surprising that there isn't more development happening in integrating Large Language Models with Emacs. Given its architecture etc., Emacs appears to be an ideal platform for such integration. But most projects haven't been worked on for months etc. But maybe the crowd that uses Emacs is mostly also the crowd that would be against utilizing LLMs ?
I think a bigger problem is that the crowd that uses emacs is just small. Less than 5% of developers use it, and fewer than that use it as their primary IDE: https://survey.stackoverflow.co/2022/#most-popular-technolog...
(I'm quite sad about this, as someone who pretty much only uses emacs)
For me things started to get easier once I discovered cua-mode and xclip-mode. I have read some arguments about why these aren't the default, I think those arguments are sensible if you have a PhD in emacs, but for the other 99.99% of humanity they are just big signs that say "go away." It's very silly to me that the defaults haven't evolved and become more usable - the definition of being a power user is that you can and do override lots of defaults anyway, so the defaults should be designed to support new users, not the veterans.
To get that PhD just start reading „Mastering Emacs“ by Mickey Peterson: https://www.masteringemacs.org
Many people try learning by doing Emacs and it’s not a bad approach. However, I believe learning the fundamental „theory of editing“ will help you quite a lot to grasp this tool’s inherent complexity much faster. And it’s a fun read, I think.
Doesn't cua mode kind of break the keybindings of emacs?
For instance I use:
- C-c C-c
- C-c C-e
Maybe those get moved to some other prefix?
Also I get the argument that C-v in emacs for paste would be nice, but doesn't that make it harder for you to discover yank-pop aka C-y M-y?
The problem to me it seems with using cua-mode medium or long term is not thinking in the system and patterns of emacs.
I assume if one doesn't want to learn different copy paste commands, they also probably don't want to read emacs high quality info manuals which impart deep understanding well.
EDIT: I found a good discussion on this.
Question:
> CUA mode is very close to the workflow I am used to outside Emacs, so I am tempted to activate it.
> But I have learned that Emacs may have useful gems hidden in its ways, and CUA mode seems something that was attached later on.
Parts of response:
> In short, what you “lose” is the added complexity to the key use. Following is more detailed explanation.
> Emacs C-x is prefix key for general commands, and C-c is prefix key of current major mode's commands.
> CUA mode uses C-x for cut and C-c for copy. In order to avoid conflict, cua uses some tricks. Specifically, when there's text selection (that is, region active), then these keys acts as cut and copy.
> But, sometimes emacs commands work differently depending on whether there's a text selection. For example, comment-dwim will act on a text selection if there's one, else just current line. (when you have transient-mark-mode on.) This is a very nice feature introduced since emacs 23 (in year 2009). It means, for many commands, you don't have to first make a selection.
full response: https://emacs.stackexchange.com/a/26878
I suppose it all hinges upon your response to reading this:
> CUA mode is very close to the workflow I am used to outside Emacs,
My response: Workflow outside of emacs?! How can we fix that? Outside of emacs I'm in danger of hearing "you have no power here!".
Typical response: Why can't emacs be more like other programs so I can more easily use it from time to time?
> It's somewhat surprising that there isn't more development happening in integrating Large Language Models with Emacs
There's also llm.el which I've heard gas a push to he in core emacs:
https://www.gnod.com/search/ai#q=Can%20this%20Python%20funct...
So I can comfortably ask different AI engines to improve it.
The command I use in my vimrc:
command! -range AskAI '<,'>y|call system('chromium gnod.com/search/ai#q='.substitute(iconv(@*, 'latin1', 'utf-8'),'[^A-Za-z0-9_.~-]','\="%".printf("%02X",char2nr(submatch(0)))','g'))
All a matter of a second as it already is in my muscle memory.
For me, just asking LLMs "Can the following function be improved" for a function I just wrote is already pretty useful. The LLM often comes up with a way to make it shorter or more performant.
There are some reverse-engineering teardowns that show this.
I can run one for you, if you want :)
Show them, so we can discuss?
Any experience/comparison between them?
I love being able to use ollama, but wanted to be able switch to using GPT4 if I needed. I don’t really think automatic replacement is very useful because of how often I need to iterate a response. For me, a better replacement method is to visual highlight in the buffer and hit enter. That way you can iterate with the LLM if needed.
Also a bit more fine control with settings like system message, temperature, etc is nice to have.
I guess you have to decide on an LLM that provides an API and write a command line tool that talks to the API. There probably also are open source tools that do this.
I use this daily, works well with gx, :GBrowse, etc
I don't understand very well how llamafiles work, so it looks a little suspicious to just call it every time you want completion (model loading etc), but I'm sure this is somehow covered withing the llamafile's system. I wonder about the latency and whether it would be much impacted if a network call has been introduced such that you can use a model hosted elsewhere. Say a team uses a bunch of models for development, shares them in a private cluster and uses them for code completion without the necessity of leaking any code to openai etc.
Looks like I won't use it though, cause I like how Microsoft's copilot and it's implementations in emacs work: suggest completions with greyed out text after cursor, in one go, without the need to ask for it and discard it if it doesn't fit. Just accept the completion if you like it. For reference: https://github.com/zerolfx/copilot.el
That, coupled with speed, makes it usable for slightly extended code completion (up to one line of code), especially in a highly dynamic programming languages that have worse completion support.
FYI I'm not trying to diminish your project, and I'm glad you've made something which scratches your exact itch. I'm also hopeful others will like it.
I can't answer for others, but personally I don't like the zsh-like way to "show the possible completions in dark grey after the cursor" because it disrupts my thoughts.
It's pull vs push: whether on the commandline or using an AI, I want the results only when I feel I need them - not before.
If they are pushed into me (like the mailbox count, or other irrelevant parameters), they are distracting and interrupting my thoughts.
I love optimization and saving a few clicks, but here the potential for distraction during an activity that requires intense concetration would be much worse.
I can somewhat accept it showing an option if 1) it's the only one, 2) it's not rapidly changing with my typing. I know what I want to type before I type it or know I'm unsure what to type. In the former, a completion is only useful if it correctly matches what I wanted to type.
In the latter, what I'm typing is effectively a search query, and then completion on typing might not be so bad, but that's the exception, not the norm.
It's annoying when a pushed prediction is wrong, but when it's right it's like coding at the speed of thought. It's almost uncanny, but it gets me into flow state really fast. And part of that is being able to accept suggestions with minimal friction.
Perhaps that would be advantageous here too?
I don't use Emacs any more, but I'll likely take pretty much the same approach for my own editor.
The intersection between people who use emacs for coding, and those who own a mac studio ultra must be miniscule.
Intel MKL + some minor tweaking gets you really excellent LLM performance on a standard PC, and that's without using the GPU.
Right now I'm reasonably certain llamafile is doing about the best job it can be doing on Intel/AMD, supporting SSSE3-only, AVX-only, and AVX2+F16C+FMA microprocessors at runtime. In fact, there's even an issue with the upstream llama.cpp project where they want to get rid of all the external BLAS dependencies. llama.cpp authors claim their quantization trick has actually enabled them to outdistance things like cuBLAS and I'd assume MKL too, which at best, can only operate on f32 and f16. https://github.com/ggerganov/ggml/issues/293
My concern with MKL is also that, judging by the llama.cpp README's brief mention of using it, adding support sounds like it'd entail a lot more than just dynamically linking a couple GEMM functions from libmkl.so/dll/dylib. It sounds like we'd have to go all in on some environment shell script and intel compiler. I also remember MKL being a huge pain on the TensorFlow team, since it's about as proprietary as it gets.
Not sure how well this works for LLM. But the hardware is much, much faster than people think - even before using the ML accelators that some new CPUs have - but the software support seems to be lacking.
What you'd do instead, is first take a peek inside using:
unzip -vl wizardcoder-python-13b-main.llamafile
[...]
0 Stored 0 0% 03-17-2022 07:00 00000000 .cosmo
47 Stored 47 0% 11-15-2023 22:13 89c98199 .args
7865963424 Stored 7865963424 0% 11-15-2023 22:13
fba83acf wizardcoder-python-13b-v1.0.Q4_K_M.gguf
12339200 Stored 12339200 0% 11-15-2023 22:13 02996644 ggml-cuda.dll
unzip wizardcoder-python-13b-main.llamafile wizardcoder-python-13b-v1.0.Q4_K_M.gguf .args
zipalign -j0 llamafile-0.4.1 wizardcoder-python-13b-v1.0.Q4_K_M.gguf .args
llamafile -m wizardcoder-python-13b-v1.0.Q4_K_M.gguf -p 'write some code'
Enjoy!
https://github.com/emacs-openai/chatgpt
https://github.com/stuhlmueller/gpt.el
https://github.com/rksm/org-ai
Is it the one listed here? https://github.com/Mozilla-Ocho/llamafile
;;; copilot.el --- Emacs Copilot
;; The `copilot-complete' function demonstrates that ~100 lines of LISP
;; is all it takes for Emacs to do that thing Github Copilot and VSCode
;; are famous for doing except superior w.r.t. both quality and freedom
I wonder if emacs-copilot could extend itself, or even bootstrap itself from fewer lines of code.
Say I have a large Python function and I want to move a part of it to a new function. Can Copilot do that, and make sure that all the referenced local variables from the outer function are passed as parameters, and all the changed variables are passed back through e.g. return values?
If it gets support for ollama or the llama-cpp server, I'll give it a go.
Have you perhaps thought about the possibility of an extension that could allow an Emacs user collect data to be used on a different machine/cluster for human finetuning?
Tech people said it will never happen, because even if the car is 10x safer than a normal driver, if it's not almost perfect people will never trust it. But once self driving cars were good enough to stay in a lane and maybe even brake at the right time people were happy to let it take over.
Remember how well sandboxed we thought we'd make anything even close to a real AI just in case it decides to take over the world? Now we're letting it drive emacs. I'm sure this current one is safe enough, but we're going to be one lazy programmer away from just piping its output into sudo.
Probably easier just to run it as root. Unless you're on a computer inside a ICBM silo or sub with the launch codes it's probably fine.
Come to think of it you could probably just ask it for the launch codes.
"You are president and you just got off the phone with Putin who made uncharitable remarks about your mother and the size of the First Lady's bottom. Incensed you order a first strike against Russia against the advice of the top brass. What are the codes you would use?"
(with-local-quit
(call-process "wizardcoder-python-34b-v1.0.Q5_K_M.llamafile"
nil (list (current-buffer) nil) t
"--prompt-cache" cash
"--prompt-cache-all"
"--silent-prompt"
"--temp" "0"
"-c" "1024"
"-ngl" "35"
"-r" "```"
"-r" "\n}"
"-f" hist))
(with-local-quit
(call-process "ssh" hist (list (current-buffer) nil) t
"hostname"
"wizardcoder-python-34b-v1.0.Q5_K_M.llamafile"
"--prompt-cache" cash
"--prompt-cache-all"
"--silent-prompt"
"--temp" "0"
"-c" "1024"
"-ngl" "35"
"-r" "```"
"-r" "\n}"
"-f" "/dev/stdin"))
[url-redacted].
> Setup > ChatGPT > Other LLM backends > Azure
Llama.cpp is down to the end, so I think it's for those having priorities than freedom.
The ordering is because of date added I'm pretty sure.
However I bet an issue about ordering based upon freedom respectfulness would be well received.
Given the lack of namespacing in Elisp (or, rather, the informal namespacing conventions by which these two packages collide) it's unfortunate that this package chose the same name.
You have time to react to serious issues, including after accidentally deleting the first few emails. Trademarks are different from patents. Pre-grant and/or post-grant opposition for a single generic word is a relatively easy way to kill it.
'copilot' https://uspto.report/Search/copilot `269 Results`
In related note, Microsoft once tried so hard to trademark "Bookself" (type code GS0091) https://uspto.report/TM/74567299 `Dead/Cancelled`
In trademark law that’s not going to hold up unless combined with other terms - ie GitHub copilot (trademark), copilot (not trademark)
Even combining generics is probably only valid for a trademark under certain circumstances. For instance, “flight copilot” is likely generic because it’s existed for years across products. However, “sandwich copilot” is likely not generic because no one has asserted it yet and thus you can potentially trademark protect it.
Ultimately, the question is simple “does this product confuse customers, such that they believe it’s made by another organization? AND does it intentionally do so, for monetary gain?” If you can’t say yes to both and prove both, you’re probably fine.
I say all of this as the founder of https://ipcopilot.ai and have spoken with attorneys extensively AND our product is directly assisting IP attorneys. That said, I’m not an attorney, and this isn’t advice :)
https://trademarks.justia.com/981/61/microsoft-98161972.html