Hacker News Reader

It's amazing how much misinformation and vague information there is on this topic. I tried getting to the bottom of this in the following post in the OpenAI forum:

https://community.openai.com/t/fine-tuning-myths-openai-docu...

Bottom line is that fine-tuning does not seem to be a feasible option for adding new knowledge to a model for question answering.

My research shows otherwise. Tuning via transformer adapters pretty much added new knowledge to QA models or could be used for adversarial QA training. You can throw away learned adapters anytime and retrain from scratch with new information if your adapters become stale. Fine-tuning is cheap and small (e.g. 60kB data in an adapter). You can customize it in production for each individual customer as well by swapping adapters at the time of inference. Embeddings for very short-term facts and adapters for medium-long-term info seems like the best combination.

You mean like what's described in this blog post, correct?

https://adapterhub.ml/blog/2022/03/adapter-transformers-v3-u...

Yes, those adapters. There are many types now, most recently LLaMA adapter:

https://arxiv.org/abs/2303.16199

Could you link to your research and/or describe the models, libraries, data and tests you used for this?

Have you tried fine-tuning via adapter, if so what has been your experience and was the the total cost?

I liked this video:

https://youtu.be/9qq6HTr7Ocw

"OpenAI Q&A: Finetuning GPT-3 vs. Semantic Search - which to use, when, and why?"

I can't determine if this person is knows what they are talking about or is an extreme amateur just making things up and speaking confidently. He has other videos that are complete nonsense.

I mean... I you aren't OK with that, how are you going to ever use a large language model? ;P

The search+prompt approach has another benefit, which is that allows a chat interface to "cite it's source", something you really can't do with just parameters (fine-tuned or not.)

Although a lot of people are working on hallucination reduction, I think we're a long way off from that in the general case. So having the ability to point to a real piece of data, outside the model, is important for applications where accuracy matters.

I've done that with the rough Q&A tool I made. I include links to websites or documents not so much for verification, but because I work with government info and the devil's in the details. The Q&A is more like a smart search engine, with the summarized answer being a nice bonus.

I have similar questions for code assistance.

Github Copilot seems to be the most effective code assistant currently. It seems to use many heuristics for figuring out relevant snippets to include in prompts, like computing Jaccard similarity of windows of the last 20 opened files. It also tries some tree-sitter cleverness for some languages, but when I snoop on the HTTP traffic it seems to almost always just give up and only include the rest of a file as context.

I have wondered whether a model fine-tuned on my own code would do much better, and be simpler. But perhaps building embeddings and searching them (like in the article you linked) would be superior.

Code assistants need to be super low latency though which maybe complicates things too.

This is why Tabnine got e super excited, until I ran across the issue where they think their results are better than what the IDE gives you, which is incredibly annoying. https://github.com/codota/tabnine-intellij/issues/18 . I would be happy to pay, but it seems they are convinced their way is best.

I honestly think that if you could have all your private code indexed and accessible, this would be a game changer as it has way better context.

I think so too. I’m thinking about writing a little server you would run as a sidecar process. It would tokenize your code and build a local index in a vector DB. Client plugins in IDEs would ask that server for similar code snippets, with a short deadline for response, perhaps 1ms. Then the plugins could assemble a prompt to send to a configurable LLM backend, and do editor-specific code insertion or suggestion or whatever. I use emacs, so I know how I want that to work for me, but it must be different for others.

My open questions though are:

- does the tokenizer for the index need to be the same as the one used by the LLM?

- is running a local LLM feasible, or should I just assume prompts must sent over the network? This would affect latency and privacy a lot.

- is fine tuning helpful? Or necessary, even, to ensure the LLM only provides source code, and only in the target language?

What are you using to decrypt the TLS layer in your snooping?

You can disable TLS cert verification when going through a proxy in VSCode. There is an option to disable “strict SSL”. Then I run a man-in-the-middle HTTP proxy and capture the traffic.

Also possible when using the Vim copilot plugin (or its emacs port) which is pretty informative - you can see a lot of the extra tricks the VSCode implementation uses.

Proxyman is a fantastic piece of software for this on Max, there’s a free version which can snoop on up to 6 apps at a time (it installs a root certificate)

I would really like to have something local, without running into ChatGPT limits or costs. I looked at some text generation models, but I couldn't find any where you can ask a question and it generates the correct code.

Does anybody know if this is already available on Hugging Face or somewhere else?

I'm using TypeScript, so my idea would be to finetune on all used packages, maybe even all of the documentation as well.

I've created a small app where I create a prompt for an eslint error and let ChatGPT come up with a git diff. I manually feed that back to my application, it eslints and compiles and if there are any errors, a new prompt is generated. This works quite well.

GPT-4 is not available for me in the OpenAI API, so it's quite cumbersome.

The best option right now is SuperCOT 30B, a LoRA for LLaMA trained on Chain-of-Thought coding questions and answers. You will need at least 24GB of VRAM to load the 4bit GPTQ quantized LoRA merged model* locally. https://huggingface.co/tsumeone/llama-30b-supercot-4bit-128g...

*The merged model contains the LoRA. Applying the LoRA over a "raw" llama model uses more VRAM and does not allow for full context in 24GB of VRAM.

An answer from OpenAI will be very biased. OpenAI will happily take your money to query their models with long prompts. They will also happily take your money to compute embeddings and thus help you search (and lock you in!).

But, as far as I know, OpenAI will not help you fine-tune, they will not run a fine-tuned model for you, and they would probably prefer that you use their models over open models that can be fine tuned.

(None of this is to say that fine tuning is better. I’m just saying that OpenAI has a strong commercial bias.)

What about their fine tuning API? https://platform.openai.com/docs/guides/fine-tuning

Interesting. But not supported for their chat models.

I like how the fine-tuning page talks about how fine tuning is supposedly better, though.

I saw a video tutorial about it and if i understood correctly, the API will automatically include whatever you "tuned" in the prompt, making it just as costly i guess. But i could be wrong as it was late and i was getting sleepy.

> the trick where you search for relevant content and paste that into a prompt

Supabase Clippy was the first docs site to ship this experience to production as far as I can tell: https://supabase.com/blog/chatgpt-supabase-docs

I believe they called it "context injection" and I have been following suit in my own writing on the topic.

I am prototyping experiences like Supabase Clippy and am also very interested in fine-tuning for docs Q&A. But my main question is: what exactly would the fine-tuning inputs and outputs look like for docs Q&A?

Edit: For Q&A the question is the input and the answer is the desired output? Is that right?

A more general comment about fine-tuning for docs from my blog:

> AI is all about prediction. Given this temperature, this wind, this day of the year, what is the chance of rain? Temperature, wind, and date are your inputs. Chance of rain is your desired output. Now, try to apply this same type of thinking towards documentation. What are your inputs? What’s your output? The page title and code block could be your inputs. Whether or not the code builds could be your output. Or maybe the code block should be the output? This is why I keep saying that applying fine-tuning to docs is tricky. What are the inputs and outputs?

https://technicalwriting.tools/posts/ten-principles-response...

(I am an AI n00b and have not looked deeply into how fine-tuning works but it's high on my list to experiment with OpenAI's fine-tuning API. Please LMK if I am getting any fundamentals wrong.)

Which OpenAI fine-tuning API? I don't think fine-tuning is currently available for GPT-4

https://help.openai.com/en/articles/7127982-can-i-fine-tune-...

Langchain wrote about LLMs and SQL, and although it's about SQL, it's still a great read, especially the references. https://blog.langchain.dev/llms-and-sql/

I'm also super keen for the 32k API limit for OpenAI, that's going to be great.

> If you want to be able to do Q&A against an existing corpus of documentation, can fine-tuning an LLM on that documentation get good results, or is that a waste of time compared to the trick where you search for relevant content and paste that into a prompt along with your question?

If you want exact detailed recall, then using a framework that provides search and recall (by embeddings or otherwise) is probably always going to beat fine tuning, but also remember, it doesn’t have to be either-or.

I mean, if you want a person to handle Q&A on a corpus, is it better for them to have studied it, or have direct access to the corpus with an appropriate index? The answer is clearly that its better if they’ve trained and have access to the corpus, and while LLMs aren’t the same as people, I think the answer for them is the same here.

As long as the search part of search + prompt is good, the prompt part will emit accurate results or will say it couldn't find the answer. You can also cite the sources this way.

It seems pretty expensive because you may be pasting a lot of context into each query. If you allow the user to have follow up queries and you want to retain context of the conversation that seems expensive too. But it does seem like it should give the best results for Q&A as long as question is directly answered in your data somewhere.

With search, you don’t know if the model/search engine will retrieve the right context. With fine tuning you don’t know if it will forget important info, or learn incorrect things.

With search, you can "tune" the query by using related references to the documents.

Ideally, we include documents that are relevant to the topics at hand and eliminate documents that aren't. Using prompt chaining, or an iterative process of labeling and extracting features, we may be able to increase the efficiency of the final prompt.

Obviously this increases costs, but so does repeated fine-tuning.