Replit's new AI Model now available on Hugging Face
blog.replit.com
blog.replit.com
These small models just suck compared to the larger ones.
It get it, they’re quick and cheap (ha! Relatively) to make and good for research and fine tuning…
…but can anyone here speak authoritatively on fine tuning and getting good results out?
I’ve been super disappointed by how bad even the q6 13B code llama model is at generating consistent code (Ie. It even compiles, forget doing what you asked) > about 30 lines in length.
These smaller model seem good for a line or two, maybe, but gosh… it’s an effort to do anything useful with them out of the box.
Carefully crafted prompt.
Tests, hand written.
Iterate: prompt, compile, run tests, generate code metrics. Accept code that passes the tests and beats the target threshold.
You’re looking at like 7-10 iterations per prompt to get anything for simple (< 30 lines) functions, and maybe no candidates after 30 iterations for longer complex requests.
Are people just using this for 5 line code snippets and autocomplete?
Or is there a way to get better results by fine tuning?
I've had the same experience as you. Quantitative, these models are decent. Qualitative, in my daily work, they're not good at all! We'll need better tests.
It seems that the general wisdom around LLMs is that you can get very good performance on small models if you fine tune for a specific task. In the case of code generation, I think you might get a good performance by fine tuning it on a specific programming language + codebase or architectural pattern.
The main problem with fine tuning is getting a good dataset, so a cheap alternative would be to put a few examples of what you want to generate in the prompt. Then you would save these prompts as task specific "fine tunes" that you would select when you need to accomplish something.
You might find this discussion helpful: https://news.ycombinator.com/item?id=37813806
It would seem so, but is there any stories/research done that actually proves that someone has done so with good results? FOSS of course, so one could actually inspect there is no fudging and so on.
I've managed to run Codellama instruct 13b with my laptop's RTX 3070 (8gb VRAM) at 6tk/s by offloading 27 layers into the GPU with llama.cpp
I've been considering getting a macbook for running 34b+ LLM inference, but with the speed in which small LLMs are progressing, I think it is better to get a laptop with an RTX 4090 and 16gb vram. Maybe It can run 34b models by offloading layers into the GPU.
...
> It’s better to make draft and bounce ideas with it than to ask it to start from scratch.
Mm. Look, I'm going to be brutally blunt here. In the long term, chat is an AI-anti-pattern.
You can't automate a prompt sequence when the Nth prompt is context dependent on the previous prompt.
"Write me XX" ... "No, fix this" ... "no, more like this" ... "I get this error" ... Cool. You get a result and it works.
...but how many interactions did you do to get that? 5? How long did it take? Did you even try 'regenerate answer' and look at some variations? Are you sure the first answer it gave you was the best one? I'm pretty sure it wasn't.
Anyway, ok, so now you have 50 functions you need to generate. Now you have 500. What's your plan? Same thing?
There are too many human touch points.
You know what AI superpower is? Automation. Repeatedly generating output, day in and day out. That's what computers are all about.
Don't get me wrong; the interactive style of AI copilot is lovely too, but it's just an incremental improvement on autocomplete, and I'm not interested; I already have autocomplete.
> how do you use it?
1) Every code function I want to generate, I create a scaffold that defines the exact function template, like:
// Using these imports only
import {x, y, z} from "./blah";
/* What does foo do... */
export function foo(a: number, b: number) { ... }
2) I create a test harness that defines a set of unit tests that define the behaviour of foo.
So, you can literally run: `npx jest ./output/foo.ts`
Every prompt has a matching `tests/foo.test.ts` test file.
(Yes, I know this sounds like a pain in the ass, it's less annoying when you scaffold tests out an LLM as well. It's not as bad as you might imagine once you get used to the workflow).
3) I process the prompts folder, and for every prompt generate a solution candidate:
- I extract the typescript from the markdown output, save it.
- I run `npx tsc --strict foo.ts --outDir dist` on it.
- If it fails, run a meta 'fix this typescript with these errors' prompt over it.
- I run the test suite on the result if it passes.
- If the test suite passes, I save the result as a candidate solution.
- If it fails, I vary the temperature and generate a new solution.
- Eventually if I don't get any candidate solutions, I log an error to revisit and refine the prompt.
Look, it's not magic, it's very simple:
LLMs generate code. sometimes the code is good, sometimes its not... but you can generate 10 or 20 different variations and it costs literally nothing except time. You just repeat it over and over and over; and maybe run some automated fixes on the outputs.
It works fine. I've made a raytracer with it, I've made a little card game with it. I'm building a website with it. Great stuff.
...if I use the openai api.
Now, the openai api sucks for lots of reasons, but the big one is that when you use the real AI superpower; ie. automation, it actually starts costing you a not insignificant amount of $$$.
So, I've been experimenting with using some offline models; specifically, as I said, code llama, and mistral. The best results I've had are from the q5, q6 codellama (1) 34B model, running using llama.cpp.
It's just slow.
So, I was experimenting with these smaller models, but... they're not that great for what I'm doing.
What you're doing, is not what I'm doing, and not quite I'm trying to do.
I get the "you're using it wrong" argument, yup. Fair enough. You're totally right. A lot of people get a lot of value from just having chatGPT open side-by-side with vscode. That's cool... but I'm specifically talking about my difficulties with a different use-case.
[1] - https://huggingface.co/TheBloke/Phind-CodeLlama-34B-v2-GGUF
Please give Ollama a go! Would love to hear if it works out! Feel free to contact my email in my profile if you need some help.
It's pretty obvious if you're writing prompts you have to use the correct prompt syntax.
? ollama seems unrelated to the problems I'm having.
There's no way you can define an arbitrary mapping between prompt formats where some have eg. SYSTEM and some don't. It's simply not possible. You have to update your prompts for different models.
I keep a separate list of prompts for each model. It's no big deal.
Agreed on prompts. We are doing a lot to guide it, and even autorepair loops. Likewise, keeping the interaction model to generating small code likewise helps the chance of any individual step being right and repairable..
It sounds like OP is trying to replace junior/mid-level SWEs with CodeLLMs where a detailed description of the desired solution goes in, and working code comes out - all hands-off.
If there is ever will be a time that LLMs can consistently achieve what OP wants, there will be a reckoning in software engineering. It's not like junior SWEs aren't already having a hard time with the current hiring environment.
Python is their forte, JS is okay. Rust is a mess, they don't get borrowing.
The best so far was early chatgpt4 but it has since been nerfed down considerably.
It's good to get ideas and for writing algorithms you are too lazy to google and implement, not great at doing actual work.
Funnily enough, I think they'd do great at FANGs interviews
> Replit intends this model be used by anyone as foundational model for application-specific fine-tuning without strict limitations on commercial use.
> The model is trained specifically for code completion tasks.
Nice, I expected that I would need to give my E-Mail address to them and that it would be ""free"".
This blog post doesn't mention FIM either, so I guess that's still missing? The demos I've seen of Replit Ghostwriter indicate that it is still possible to get good results without FIM, as long as you have good enough software around the model, but I think FIM could still improve things further.
The much smaller Refact-1.6B model supports FIM[1], and Refact-1.6B worked pretty well when I tested it a few weeks ago.
People (like the most upvoted comment in this thread) who are expecting any of these small models to write entire programs for them based on a simple prompt seem to misunderstand the purpose of these smaller models, which is to be a smarter alternative for code completion. Writing entire functions or programs is better suited to much larger (and slower) instruct/chat-tuned models.
[0]: https://huggingface.co/replit/replit-code-v1-3b/discussions/...
[1]: https://refact.ai/blog/2023/introducing-refact-code-llm/
Any idea where the list can be found?
Following the link to the "Stack Dedup V2" page: https://huggingface.co/datasets/bigcode/the-stack-dedup
> The Stack contains over 6TB of permissively-licensed source code files covering 358 programming languages. The full list can be found here.
https://huggingface.co/datasets/bigcode/the-stack-dedup/blob...
It requires login to see the JSON file.
Didn’t Meta say they trained on 2k A100s for LLama 2?
Tech talk here with timestamp: https://www.youtube.com/live/veShHxQYPzo?si=UlcU9j2kC-C4oWvj...
Roughly $1/hr/GPU in power cost so looking at 128247 = $21,504.
Cheap compared to OpenAI, but not something an indiehacker can do by themselves unless they have millions to burn.
Source: https://huggingface.co/replit/replit-code-v1_5-3b
I'm not an ML engineer, just interested in the space - but as a general ballpark, training these models from scratch needs hundreds to thousands of GPUs.
Surely someone is working on general AI powered code analysis tooling?
I don’t consider cloud inference to count as getting it working “locally” as requested by the comment above yours.
Refact worked nicely and worked locally when I tried it a few weeks ago, but the challenge with any new model is making it be supported by the existing software: https://github.com/smallcloudai/refact/
You can use the Hugging Face Inference API or your own HTTP endpoint, provided it adheres to the API specified here[1] or here[2]."
It's fairly easy to use your own model locally with the plugin. You can just use the one of the community developed inference servers, which are listed at the bottom of the page, but here's the links[3] to both[4].
[1]: https://huggingface.co/docs/api-inference/detailed_parameter...
[2]: https://huggingface.github.io/text-generation-inference/#/Te...
[3]: https://github.com/wangcx18/llm-vscode-inference-server
[4]: https://github.com/wangcx18/llm-vscode-inference-server
WizardCoder 34B and Phind 34B are the only models remotely comparable, and they are still slightly worse than GPT 3.5 (let alone GPT4).
https://wandb.ai/byyoung3/ml-news/reports/Fine-Tuning-Mistra...
What would be interesting is compare GPT4 at a certain task with a small model fine tuned for that task.
Even if each inference pass only runs part of the network, there's still a trillion learnable parameters there lol.
So comparing it with big models I'd say it's good but might have limited usefulness
(you can probably go further with 3B with only code)
The table just below this shows the other models doing better on half of the benchmarks; the Replit column being in boldface is misleading.
I just thought it was meant to draw attention to their numbers.
I think colouring a column is the common approach to drawing attention to your own while still respecting the best is bold custom, which they've sort of done with the header, but personally I'd have gone with the cell background for the column.