joaquincabezas

Now that many open source LLMs are already “useful” it is key to ease the development _around_ them, in particular in a way where users-developers can leverage on “private data” (really, data that is not present in the pre-training of these models).

Even if in the repo it is stated that it is optimized for big models (needing A100/H100), I still feel this can help with smaller models more than large ones.

We can extend “If you build it, they will come” with “If you provide the tools, the will build”

I’ve already heard a few times that the selection of models is seen as a problem (need to benchmark continually, justify changes…), this is an elegant solution.

I don’t know if choosing different models for the same consumer can be problematic (seen as not consistent), but maybe using this approach will force the post-processing code not to be “coupled” with one particular model.

I tend to see a price range as a trick to make people think that the lower price is a good deal (hey look today I saved $250M!!)

LLM inference optimization has been key for the OpenAI GPT-4o presentation (2x faster, 50% cheaper) and its driving lots of industry research because it’s direct cost savings, but it’s refreshing to see so many techniques published as papers (i.e from Stanford, Berkeley…)

I normally use Netron for quickly inspecting models and making this 'mental picture' of the architecture, but this hierarchical approach seems a better fit for my needs.

I'm just starting and the first impression is pretty good!

wow their graphs at the GitHub README (https://github.com/HazyResearch/ThunderKittens/blob/main/att...) make me extremely dizzy. Are these wavy bars even legal? :P

I was at ICLR and this was one of the best of this year, it was also evident during the poster session. Congrats to the authors!!

During the first year of my PhD I tried building metrics for interpretability of Graph Neural Networks… kind of obsessed about having “measurable” properties because I thought that was the “science way”.

I failed every single try, and after some time I began appreciating qualitative indicators.

Also, Chris Olah’s articles are lovely (Anthropic and also distill.pub ones)

it looks like TensorRT-LLM (TRT-LLM) is the way to go for a realtime API for more and more companies (i.e perplexity ai’s pplx-api, Mosaic’s, baseten…). Would be super-nice to find people deploying multimodal (i.e LLaVA or CLIP/BLIP) to discuss approaches (and cry a bit together!)

this keeps coming… recently there’s been a bit of debate between Milvus and Qdrant on this topic: https://www.linkedin.com/posts/yujiantang_how-can-you-judge-...

PaddleOCR works pretty well, how are you planning to integrate it in your workflow? I found huge differences in throughput between python serving and frameworks (i.e. NVIDIA Triton Inference Server).

are there also experiments around image embedding generation to use in combination with the LLM?

maybe for this use-case is it better to execute the vision tower on a GPU and leave the LPU for the language part?

Also discussed at: https://news.ycombinator.com/item?id=38870249

and without ChatGPT…

Thanks a lot for the material Varun, neat presentation with exhaustive computations that make it easy to follow. Question on the serving part: vLLM, Deepspeed, TensorRT-LLM... ? Thanks!

imagine being in the last round of interviews for joining OpenAI…

Totally recommend that book!!

I found this a few years ago at https://stackoverflow.com/questions/14754762/can-gitconfig-o... and has been my preferred setup.

I was always curious about the original cells of the parents (or more specifically the molecules forming it). Are they somehow “staying” in the baby?

reading frequently on HN how COBOL is still running critical tasks, I would rather say "for fun and profit"

Is this the real life? Is this just fantasy?

> and found myself nodding my head in vigorous agreement.

s/he who is without sin…

Our IoT device was not selling well because it lacked one RS-485 port… and then a 4-20mA interface, and then…

Did you finally select it? Our team is using PaddleOCR with good results! We wrote an article (https://medium.com/adevinta-tech-blog/text-in-image-2-0-impr...) about it.

all you need is emergent anything

fn main() { println!("Too late I guess!"); }

I keep saving these Rust resources for a near future... Am i the only one??

I really hope to start using Rust in 2023, probably for some kind of API gateway experimentation

Check https://logconference.org/schedule-tutorials/#neural-algorit... for a tutorial happening in less than 24 hours.

LoG conference is free to attend and online :)

I am really curious about Rust after looking for API proxies (in particular Linkerd2 - https://github.com/linkerd/linkerd2-proxy). Can anyone share her/his experience on going from Python to Rust? (bonus point if related to API :lol:)

I remember a long time ago, reading about Semantic Web and intelligent agents and dreaming of a Natural Language interface for planning journeys…

“I want to travel from Seville to Berlin next October, avoiding weekends, for a two or three nights stay in a hotel by the river. Direct flights preferred.”

calling it “The Merge” (with capital letters) makes it sound like a blockbuster action movie. I’ll say I prefer the book.