1,526 karma · joined October 28, 2019
Note LoRa fine tunes aren't the same as the original Alpaca, LoRa results in some performance loss (although how much isn't clear)
Quantization isn't the only technique available for downsizing a model, Llama itself is already the result of sizing the model and input data according to "Chinchilla optimality", a very recent (as in 2022) result that e.g. GPT-3 predates. The result is that Llama-13B performs in benchmarks similarly with GPT-3 175B despite the tremendous size difference. There are separately also a variety of pruning methods to further eliminate inactive weights present in the trained model (I think this is also active research)
Finally even on something like a Raspberry Pi, implementations for inference (like llama.cpp) are nowhere near mature yet. There are already a multitude of runtimes available for inference making large tradeoffs between performance and flexibility (e.g. many models running on PyTorch vs ONNX report 5-10x speedups running under ONNX)
I think the really exciting part of Alpaca is the size and budget of the team - 5 students with $100 scraping OpenAI put this model together in a couple of hours of training. Any notions of premium persisting in the AI space for much longer seem fantastic at best, for all intents and purposes it has already been commoditized. And that's scary considering the size of the dent ChatGPT has put in my Google traffic
This pricing model seems fair since you can pass in huge prompts and request a single word reply, or a few words that expect a large reply
Seems model sizing, compression and quantization are still an art form, see also https://www.unum.cloud/blog/2023-02-20-efficient-multimodali...
Of course still relying on google in the background, but increasingly rarely, and presuming all the negative commentary we've been seeing online are folk who simply haven't tested it in anger yet. Today's chatgpt hallucination is yesterday's Google blogspam etc. Folk for some reason continue to act like the old world was perfect. This is much closer to perfection than anything we ever had, and infinitely more comprehensive. Google as we knew it is already dead, because the medium google was built for just got made obsolete. This is far closer to a new Internet iteration (WAIS, FTP, Gopher, HTTP, Web2.0, ...) than it is a new search engine
Now watch as the search engines try to adapt it to their recency-biased ads model and fail miserably, as what we have is already better than what they were able to sell. Very unclear bing or Google or anyone you've heard of will win this round, its suddenly a very exciting time in tech again
Another aspect I find very exciting is that these effectively represent a return to a curation-driven Internet, selection of input data for model training is probably an interesting new form of diversification. Who cares about having a site in the world wide web if its not part of the inputs for the language models used by millions of users? That's a completely new structure for the dissemination of ideas, marketing, "SEO" etc., and a brand new form of mass media
This quote is 18 years old, and already by then it was established understanding that Google had extreme technical magic the rest of the world lacked. A rushed, bumbled tech demo in an area a company who for decades has invested billions in and widely enjoyed a perception of AI supremacy is a wildly different thing to a polished demo from a company nobody expects anything good to come from.
They have constantly encouraged the belief in their technical dominance in their marketing and hiring for as long as I can remember. If that's the perception externally, imagine how much more scathing a defeat it must feel to current employees.
Responses on Twitter are horrific: https://twitter.com/googleeurope/status/1623251768099409920
This is the first non-technical Google presentation I've felt compelled to archive
I think folk are unestimating the scale of the change in medium that ChatGPT represents, it's not simply a problem of popping out a compelling clone for Google, they must also preserve or replace the lost ad revenue which represents 80% of their income, otherwise things quickly start to look grim. In contrast, ads are only 36% of Microsoft's income. If it turns out that llms are only economical as a subscription offering, Microsoft are already better placed to e.g. bundle NewBing with its 365 subscriptions.
I think subscription based search might be the scariest outcome, it would be something like an iPhone moment, where the entire medium becomes stratified into commodity/ad supported/low income user vs. premium/ad free/high value users. Imagine Google's legacy as the spam riddled search engine low income folk use, filled with ads for payday loans, used car dealers and predatory personal injury lawyers
The technique sounds messy but actually it involves not much work when the target is Linux, a single half-open SYN is good for 63 seconds with the default sysctl settings, which seem to be used with almost every Internet service you might want to reach (including e.g. Google)
I was playing with this during an interview last year and intended to write it up, but never got around to it. The technique seems to work as intended, I made a little prototype reverse proxy for it in Python using a temporary listening socket with a drop-all SO_ATTACH_FILTER to allocate a port number and prevent Linux on the initiator side from responding with a RST to ACKs for a half-open connection it knows nothing about
The basic gist seems to be that it's just not currently the upstream developer's focus
Nice to meet another used hardware buyer. I think the total cost of my laptops over the past 10 years equates to roughly the price of a single new high spec Macbook, and that's with repeat replacements due to damage / spills / etc.
Maybe it's possible to game archive.ph, but seems more likely this person's humor has a consistent strongly misogynistic theme. Very happy to see a site causing actual suicides taken down, but the champion of the campaign doesn't come across as any kind of hero
It's also interesting to juxtapose comments here against the fact it was so easy to find the original thread on Google.