Hacker News Reader

Comment by t_mann | Hacker News Reader

In some way, I'd be grateful if they screwed up ChatGPT (even though I really like to use it). The best way to be sure that no corporation can mess with one of your most important work tools is to host it yourself, and correct for the shortcomings of the likely smaller models by finetuning/RAG'ing/[whatever cool techniques exist out there and are still to come] it to your liking. And I think having a community around open source models for what promises to be a very important class of tech is an important safeguard against SciFi dystopias where we depend on ad-riddled products by a few megacorps. As long as ChatGPT is the best product out there that I'll never match, there's simply little reason to do so. If they continue to mess it up, that might give lazy bums like me the kick they need to get started.

View on HN

> for what promises to be a very important class of tech

What I see here is the automated plagiarism machine can't give you the answer only what the answer would sound like. So you need to countercheck everything it gives you and if you need to do so then why bother using it at all? I am totally baffled by the hype.

For things that are well covered on stack overflow, it's a strictly better search engine.

eg say you don't remember the syntax for a rails migration, or a regex, or something you're coding in bash, or processpool arguments in python. ChatGPT will often do a shockingly good job at answering those without you searching through random docs, stack overflow, all the bullshit google loves to throw at the top of search queries, etc yourself.

You can even paste in a bunch of your code and ask it to fill in something with context, at which it regularly does a shockingly good job. Or paste code and say you want a test that hits some specific aspect of the code.

And yeah, I don't really care if they train on the code I share -- figuring out the interaction of some stupid file upload lib with aws and cloudflare is not IP that I care about, and i chatgpt uses this to learn and save anyone else from the issues I was having, even a competitor, I'm happy for them.

For a real example:

> can you show me how to build a css animation? I'd like a bar, perhaps 20 pixels high, with a light blue (ideally bootstrap 5.3 colors) small gradient both vertically and horizontally, that 1 - fades in; 2 - starts on the left of the div and takes perhaps 20% of the div; 3 - grows to the right of the div; and 4 - loops

This got me 95% of where I wanted; I fiddled with the keyframe percents a bit and we use this in our product today. It spat out 30 lines of css that I absolutely could not have produced in under 2 hours.

> This got me 95% of where I wanted

Exactly. Even when it gives an answer that contains many mistakes, or doesn't work at all, I still get some valuable information out of it that does in the end save me a lot of time.

I'm so tired of constantly seeing remarks that basically boil down to "Look, I asked ChatGPT to do my job for me and it failed! What a piece of garbage! Ban AI!", which funnily enough mostly comes from people that fear that their job will be 100% replaced by an AI.

And so now nobody is adding anything new to Stack Overflow, and thus ChatGPT will be forever stuck only being able to answer questions about pre-2024 tech.

Sometimes the big picture is enough, and it doesn't matter if some details are wrong. For such tasks ChatGPT and LLM's generally are a major improvement over googling, and reading a lot of text you don't really care about that much.

Well creativity can be used outside of acaedmia, so no checking required there aside from intellectual property?

Why do we need textbooks if they are just plagaries of the original papers anyway?

You don’t need textbooks. Most textbooks are garbage.

So are most papers.

textbook have higher requirement. papers offer probably truth, but textbook should offer most important common sense in specific discipline.

It’s telling that comments like these hit all the same points. “Plagiarism machine”, “convincing bullshit”, with the millions of people making productive use of ChatGPT belittled as “hype”, all based purely on one person’s hypothesis.

The proof is in the puddling. I am far from being alone in my use of LLMs, namely ChatGPT and Copilot, day-to-day in my work. So how does this reconcile with your worldview? Do I have a do-nothing job? Am I not capable of determining whether or not I’m being productive? It’s really hard for me to take posts like these seriously when they all basically say “anyone that perceives any emergent abilities of this tech is an idiot”.

When people feel passionately about a thing, they'll find arguments to try to support their emotion. You can't refute those arguments with logic, because they weren't arrived at with logic in the first place.

Tell me how that works. You add industrial strength gaslighting to your work and not afraid of being fired...?

The truth is that we doubt that you are actually doing any productive work. I don't mean that as a personal insult, merely that yes, it's likely you have a bullshit job. They are extremely common.

For many thing I'm trying to find out, I'll have to verify them myself anyway, so it's only an inconvenience that it's sometimes wrong. And even then, it give you a good starting point.

Who are these people that go around getting random answers to questions from the internet then blindly believing them? That doesn't work on Google either, not even the special info boxes for basic facts.

> Who are these people that go around getting random answers to questions from the internet then blindly believing them?

Up until relatively recently, people didn't just vomit lies onto the internet at an industrial scale. By and large if you searched for something you'd see a correct result from a canonical source, such as an official documentation website or a forum where users were engaging in good faith and trying their best to be accurate.

That does seem to have changed.

I think the question we should be asking ourselves is 'why are so many people lying and making stuff up so much these days' and 'why is so much misinformation being deliberately published and republished.'

People keep saying that we're 'moving into a post-truth era' like it's some sort of inevitability and nobody seems to be suggesting that something perhaps be... done about that?

Excluding the internet, people at large have been great at confabulating bullshit for about forever. Just jump in your time machine and go to a bar pre cellphone/internet and listen to any random factoid being tossed out to see that happening.

The internet was a short reprieve because putting data up on the internet, for some time at least was difficult, therefore people that posted said data typically had a reason to do so. A labor of love, or a business case, in which these cases typically lead to 'true' information being posted.

If you're asking why so much bullshit is being posted on the inet these days, it's because it's cheap and easy. That's what has changed. When spam became cheap, easy, and there was a method of profiting from it, we saw it's amount explode.

> so then why bother using it at all?

Because it's still more efficient that way [0].

[0] https://www.hbs.edu/faculty/Pages/item.aspx?num=64700

The "open source" LLMs are already good enough for simple tasks GPT-3.5 was used for. I see no reason why they can't catch up with GPT-4 one day.

I assume you are referring to Llama 2? Is there a way to compare models? e.g. what is Llama-7b equivalent to in OpenAI land? Perplexity scores?

Also, does ChatGPT use GPT 4 under the hood or 3.5?

Actually, there have been new model releases after LLaMA 2. For example, for small models Mistral 7B is simply unbeatable, with a lot of good fine-tunes available for it.

Usually people compare models with all the different benchmarks, but of course sometimes models get trained on benchmark datasets, so there's no true way of knowing except if you have a private benchmark or just try the model yourself.

I'd say that Mistral 7B is still short of gpt-3.5-turbo, but Mixtral 7x8B (the Mixture-of-Experts one) is comparable. You can try them all at https://chat.lmsys.org/ (choose Direct Chat, or Arena side-by-side)

ChatGPT is a web frontend - they use multiple models and switch them as they create new ones. Currently, the free ChatGPT version is running 3.5, but if you get ChatGPT Plus, you get (limited by messages/hour) access to 4, which is currently served with their GPT-4-Turbo model.

I agree with your comments and want to add re: benchmarks: I don’t pay too much attention to benchmarks, but I have the advantage of now being retired so I can spend time experimenting with a variety of local models I run with Ollama and commercial offerings. I spend time to build my own, very subjective, views of what different models are good for. One kind of model analysis that I do like are the circle displays on Hugging Face that show how a model benchmarks for different capabilities (word problems, coding, etc.)

no it's mistral. mistral 7b and mixtral 8x7b MoE which is almost on par (or better than) chatgpt 3.5. Mistral 7b itself packs a punch as well.

Mixtral 8x7b continues to amaze me, even though I have to run it with 3 bit quantization on my Mac (I just have 32G memory). When I run this model on commercial services with 4 or more bits of quantization I definitely notice, subjectively, better results.

I like to play around with smaller models and regular app code in Common Lisp or Racket, and Mistral 7b is very good for that. Mixing and matching old fashioned coding with the NLP, limited world knowledge, and data manipulation capabilities of LLMs.

This is neat to know. On Ollama, I see mistral and mixtral. Is the latter one the MoE model?

yes, mixtral is the MoE model.

There is also MiQu (stands for mi(s|x)tral quantized I think?) which is a leaked and older mistral medium model. I have not been able to try it as it needs some RAM / VRAM I don't have but people say it is very good.

> Is there a way to compare models?

This is what I like to use for comparing models: https://huggingface.co/spaces/lmsys/chatbot-arena-leaderboar...

It is an ELO system based on users voting LLM answers to real questions

> what is Llama-7b equivalent to in OpenAI land?

I don't think Llama 7b compares with OpenAI models, but if you look in the rank I linked above, there are some 7B models which rank higher than early versions of GPT 3.5. those models are Mistral 7b fine tunes.

llama 2 isn't open source

Miqu (the leaked large Mistral model) and its finetunes seem to be the most coherent currently, and I'd say they beat GPT-3.5 handily.

There are no models comparable to GPT-4, open source or not. Not even close.

The opensource ones are already competitive to GPT3.5 in terms of "reasoning" and instruction following. They tend to be significantly worse in knowledge tasks though, due to their lack of parameters. GPT 3.5 is five times bigger than mixtral after all.

It's been a few months since I tested but as far as commercially useable AIs go nothing could beat GPT 3.5 for conversations and staying in a character. Llama 2 and other available clones were way to technical (good at that tho)

Reply on news.ycombinator.com