Hacker News Reader

Comment by alecco | Hacker News Reader

Better evaluation paints a bit different picture:

https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderb...

*FreeWilly2 is a Llama2 70B model finetuned on an Orca style Dataset

EDIT: actually, impressive:

                   FreeWilly2  GPT-3.5  GPT-4
    ARC               71.1      85.2     96.3
    HellaSwag         86.4      85.5     95.3
    MMLU              68.8      70.0     86.4
    TruthfulQA        59.4      47.0     59.0

So reasoning (ARC) is lagging behind, but the other evaluations are at GPT-3.5 level and closing the gap with 4.

Source for GPT-3.5 and GPT-4.0 values (but mind it might not be the same # of shots)

https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderb...

View on HN

That seems more in-line with my experience. I have been using GPT-3.5 and GPT-4 for data cleaning pipelines, and have tried to swap out LLaMA2 70B in a few of the "easier" tasks, and it hasn't performed well enough yet for any of my tasks done by GPT-3.5.

Hi! Could you please share a few words on what type of data you are cleaning using GPT? It is an intriguing idea and I would love to learn more to see if I could use a similar approach.

Youtube transcripts. It only works with single-person channels at the moment, as I haven't worked on disambiguating multiple speakers. They are very messy if they are just an auto transcription from Google. Practically unusable in most cases.

So first step in the pipeline is cleaning up the transcripts for incorrectly transcribed words or sentences. Using the context of the rest of the transcript, it is able to fix the vast majority of them. Then we add punctuation and format it with paragraphs. Then I have another check over the whole transcript for any remaining issues.

After all of that, I have a relatively clean transcript that represents the original audio very closely. From there, I am doing things like: 1. creating question/answer pairs from the transcript 2. creating a document of additional context that fills in details about what the speaker is talking about but may not have explicitly said 3. creating a summary of the transcript identifying the main purpose 4. creating a knowledge graph from the transcript with nodes and edges 5. creating an annotated version of the transcript using that knowledge graph

I plan on putting some of this data into a vector database, and some of it will be used for fine tuning LLaMA2 models on specific tasks (like knowledge graph creation, annotation using a knowledge graph, and writing using a knowledge graph to keep track of events)

That's very informative - thanks so much for the explanation!

My only experience with transcripts is in the context of transcribing short interviews. I used Whisper and it was pretty good. I mostly work with quantitative data, though.

In terms of the disambiguation of speakers, I haven't done it, but I remember blind signal separation discussed in a signal processing seminar I attended. There is also this paper, in case you haven't seen it already: https://enk100.github.io/speaker_separation/

Thanks again!

I have not read that yet, thank you!

Also, I haven't tried using Whisper for getting a transcription from the audio. I went the route of downloading the automatically generated transcripts from Youtube for a set of videos. An audio processing pipeline is definitely something I could add later though as an additional input channel for the overall pipeline.

How do you represent knowledge in your knowledge graph? Do you use an existing open source ontology?

No, I just used YAML to represent it.

Here is an example: https://gist.github.com/Tostino/f6f19e88e39176452c1a765cb7c2...

Here is the transcript that I created that knowledge graph from, and then annotated with the knowledge graph for training purposes: https://gist.github.com/Tostino/e64524437848fbb3aebe52056df8...

Edit: I am using symbolic IDs intentionally. Reason for that was this paper: https://ai.googleblog.com/2023/07/symbol-tuning-improves-in-...

Was this with or without fine-tuning?

That is with fine-tuning: https://stability.ai/blog/freewilly-large-instruction-fine-t...

I should be clear though, I didn't fine-tune on my specific tasks myself (as that is the end result of what I am doing with the pipelines outputs). I just tried the LLaMA2-chat model (which is fine-tuned), and the Freewilly 2 finetune.

LLaMA2 is far and away from GPT 3.5. Just look at HumanEval and other code generation metrics. All these GPT-4 based "chat evals" are extremely misleading and people should take it with a bag of salt.

That's what I say in my comment, OP's ranking is quite misleading.

The ranking I linked and quoted in my comment's is much better. See the About tab. It has 4 evaluations and it doesn't use GPT-4 to evaluate.

Also the top one is a tuned Llama 2. Also clarified in my original comment.

many usecases that arent code, + people can finetune llama2 with code. its a great start for a free model.

It depends on the eval, but I think it's fair to say that it's close. Here is the AGI Eval results organized into a table w/ averages (also I put in the new Hermes LLama2 13B model as well: https://docs.google.com/spreadsheets/d/1kT4or6b0Fedd-W_jMwYp...

It beats out ChatGPT in every category except SAT-Math. We definitely need harder benchmarks.

So far, there's BIG-Bench Hard https://github.com/suzgunmirac/BIG-Bench-Hard and just published, Advanced Reasoning Benchmark https://arb.duckai.org/

Reply on news.ycombinator.com