backend-dev-33

After 100+ comments discussions here are quite difficult to read. So some kind of automatic summary of the popular discussions would be great. Somebody already did it: https://extraakt.com/

P.S. I am not the author: just found the link in the Llama 4 thread.

> additional challenge of offering it over a network socket across multiple requests and doing so in a performant manner.

@kkielhofner thanks a lot! now I realize it. I see, there is even GRPC support in Triton, so it make sense.

I was just told about this thing: https://aws.amazon.com/ec2/instance-types/g4/

one NVIDIA T4 GPU, 16 GB RAM, and, this is an EC2 instance, it means "install anything" all this for $0.526 /Hour

do you see any hidden gotchas?

UPDATE: how to do the same classification task using some hosting provider with GPU?

Let us discuss it here -> https://news.ycombinator.com/item?id=41768088

Thanks @Terretta

Well, the categories I use - do not overlap at all with the list of 1092 categories in Google Content Categories.

> it handles other classifications as well

hm... I highly doubt that. First of all - I do not see API to upload list of MY categories. Second: Can somebody with Google Cloud account try it? I have no account and when creating it - it asks for credit card...

thanks for Text Embedding Inference (never heard about it before)

> you can always run a zero shot pipeline in HF with a simple Flask/FastAPI application.

Yeah, sometimes things that are right in front of your nose, you don't see them. you mean this? https://huggingface.co/docs/api-inference/index

Well, looking at Triton Inference Server + OpenVINO backend [1]...uff... as you said: "significant amount of development effort". Not easy to handle when you do it first time.

Is ONMX runtime + OpenVINO [2] a good idea ? Seems easier to install and to use: Pre-built Docker image and Python package... Not sure about performance (the hardware-related performance improvements - they are in OpenVINO anyway, right?).

[1] https://github.com/triton-inference-server/openvino_backend

[2] https://onnxruntime.ai/docs/execution-providers/OpenVINO-Exe...

is it possible to use SetFit through some command-line interface of some API?

And here Part1 as link: https://news.ycombinator.com/item?id=20256681

Well, these OSS developers created... 53 different commenting systems: https://lisakov.com/projects/open-source-comments/

Learning a new programming language or database? let's write a commenting system! )))

And Hetzner CX11 is €2.49 monthly and you have peace of mind regarding pricing (the price if FIXED).

Probably I have to rephrase the question to something like "Alternatives and side effects of Hetzner CX11 ?"

You are right. I'm talking to artist about modifying or creating a new image

I expected to see here things like negotiations and literature.

And, absolutely unexpected, I saw "Russian". 3 times.

Would you please explain?

P.S. I'm a native Russian speaker.