Hacker News Reader

Comment by exe34 | Hacker News Reader

Multimodal doesn't guarantee input and output.

> Currently, we support image, audio and video input.

Seems I'm missing something. Does this model support other inputs?

Image outputs are supported, videos I'm not sure but I don't think that's an output, just a preview of the equirectangular example, so, same question here, what does this model outputs that isn't supported?

It's a diffusion model, completely different from autoregressive attention models.

like https://github.com/ggml-org/llama.cpp/tree/master/examples/d... ?

At the risk of stating the obvious llama.cpp isn't just about LLaMa as https://github.com/ggml-org/llama.cpp/blob/master/src/llama-... someone else pointed out.

Aha I was wrong. Thanks for sharing that!

no worries, I was wrong too, it is multimodal but only for inputs apparently, so for now there seem to only be text as output but no image as output

Not all architectures are supported by llama.cpp . The GGUF format encodes the NN in a standardized way, but then you need code that can use that NN structure.

I understand that llama.cpp could only output text, last time I checked (I do not know how to find a good source for that though).

See https://github.com/ggml-org/llama.cpp/blob/master/src/llama-... , the

  enum llm_arch {

...

I haven't used llama.cpp for image generation either but I recall an issue about it. Unfortunately I can't pinpoint it now and there is the older closed issue https://github.com/ggml-org/llama.cpp/issues/4408 so unless mtmd supports also multimodal outputs out of the box safe to assume output is still limited to text.

Reply on news.ycombinator.com