visioninmyblood

LLM-based agents handle text incredibly well, but images, videos, or PDFs with visual content are hard to interpret. mm-ctx gives your CLI agent multi-modal skills.

Try it interactively in Spaces: vlm-run/mm-ctx

Readme: https://vlm-run.github.io/mm/ PyPI: https://pypi.org/project/mm-ctx SKILL.md: https://github.com/vlm-run/skills/blob/main/skills/mm-cli-sk...

mm-ctx is meant to feel familiar: the UNIX tools we already love (find/cat/grep/wc), rebuilt for file types LLMs can't read natively and designed to work with agents via the CLI. - mm grep "invoice #1234" ~/Downloads searches across PDFs and returns line-numbered matches - mm cat <document>.pdf returns a metadata description of the file - mm cat <photo>.jpg returns a caption of the photo - mm cat <video>.mp4 returns a caption of the video

A few things we obsessed over: Speed: Rust core for the hot paths Local-first, BYO model: Uses any OpenAI-compatible endpoint: Ollama, vLLM/SGLang, LMStudio with any multimodal LLM (Gemma4, Qwen3.5, GLM-4.6V). Composable: stdin + structured outputs Drops into any agent via mm-cli-skills: Claude Code, Codex, Gemini CLI, OpenClaw.

We’d love to hear your feedback! Especially on the CLI and what file types and workflows you would like to see next.

https://meta.ai/ this is where you can try it seems like the API is not publicly accessable yet. I feel they are very late to the game and do not show value to customers over other models.

The roles seems more oriented towards visual agents. Is this is similar to google vision agent but with more capabilities?

Blog: https://vlm.run/blog/introducing-orion-artifacts

Cookbook: https://github.com/vlm-run/vlmrun-cookbook

Docs: https://docs.vlm.run/agents/artifacts

Blog: https://vlm.run/blog/introducing-orion-artifacts

Cookbook: https://github.com/vlm-run/vlmrun-cookbook

Docs: https://docs.vlm.run/agents/artifacts

Meta was lacking behind on the agents space. This is a good capture but they are making crazy good offers but not turning them into killer products so far. The AI agents space is picking up in 20206. Next they will hire voice agents like ElevenLabs and Cartesia, visual Agents like VLM Run or Landing AI and then web browsing agents.

was fun generating the as well

you can chat with vlm.run to generate these assets from image generation without needing a gpu.

Modi: https://chat.vlm.run/c/bdbaf1dc-b3c2-4b8a-ad17-6e26d87475fd

Musk: https://chat.vlm.run/c/894f44ce-c366-4c93-b348-de6eebfb9f03

Ronaldo: https://chat.vlm.run/c/db5eff27-8c52-4e0b-8e4e-157df6bef278

Gomez: https://chat.vlm.run/c/02b92f23-a6fd-4380-b7a9-52ae7463c9c1

Jackie chan: https://chat.vlm.run/c/62ca5502-2066-4d5f-b994-c8845b2e72c9

Messi: https://chat.vlm.run/c/5bfda3c2-a2bf-4ca2-a874-78f1ce66edf1

Taylor swift: https://chat.vlm.run/c/d411c16a-e2b1-490c-85cc-9f86f2564a59

Obama: https://chat.vlm.run/c/c55fe1d6-853b-4113-bdaf-d828bee0da53

Trump: https://chat.vlm.run/c/84b57509-ed38-454a-9b17-85c03a247993

Kohli: https://chat.vlm.run/c/da8ffe8a-0818-4820-bdeb-b82d965ae3a4

Lamar: https://chat.vlm.run/c/ef6488c3-b0f0-4651-944e-7d0c4f98bdd1

Mrbeast: https://chat.vlm.run/c/c24d1cac-feb0-4a53-b1a2-2bbb62a7b7fb

I tried this by using an gemini visual agent build with orion from vlm.run. it was able to produce two different images with five leg dog. you need to make it play with itself to improve and correct.

https://chat.vlm.run/c/62394973-a869-4a54-a7f5-5f3bb717df5f

Here is the though process summary(you can see the full thinking the link above):

"I have attempted to generate a dog with 5 legs multiple times, verifying each result. Current image generation models have a strong bias towards standard anatomy (4 legs for dogs), making it difficult to consistently produce a specific number of extra limbs despite explicit prompts."

you want get the exact coordinated by running a key point network to pinpoint which coordinates does the next click point is you can. here I show a example simple prompt which returns the keypoint location of the next botton to click and visually localize the point with a keypoint in the image

https://chat.vlm.run/c/e12f0153-7121-4599-9eb9-cd8c60bbbd69

I agree claude and chatgpt and even gemini does a poor job in detecting and cropping into a region. Some of the simplest tasks, Qwen also is great at summerization but not into solving simple vision tasks like cropping, segmentetation and detection. Here is an examples where we compared claude, gemini, chatgpt and other frontier models for simple(and complicated) visual tasks https://chat.vlm.run/showdown#:~:text=Crop%20into%20the%20cl...

I was using this for video understanding with inference form vlm.run infra. It definitely has outperformed Gemini which generally is much better than openai or Claude on videos. The detailed extraction is pretty good. With agents you can also crop into a segment and do more operations on it. have to see how the multi modal space progresses:

link to results: https://chat.vlm.run/c/82a33ebb-65f9-40f3-9691-bc674ef28b52

Quick demo: https://www.youtube.com/watch?v=78ErDBuqBEo

The 3d viewer is slow, and might take time to load, thank you for your patience.

Modi: https://chat.vlm.run/c/bdbaf1dc-b3c2-4b8a-ad17-6e26d87475fd

Musk: https://chat.vlm.run/c/894f44ce-c366-4c93-b348-de6eebfb9f03

Ronaldo: https://chat.vlm.run/c/db5eff27-8c52-4e0b-8e4e-157df6bef278

Gomez: https://chat.vlm.run/c/02b92f23-a6fd-4380-b7a9-52ae7463c9c1

Jackie chan: https://chat.vlm.run/c/62ca5502-2066-4d5f-b994-c8845b2e72c9

Messi: https://chat.vlm.run/c/5bfda3c2-a2bf-4ca2-a874-78f1ce66edf1

Taylor swift: https://chat.vlm.run/c/d411c16a-e2b1-490c-85cc-9f86f2564a59

Obama: https://chat.vlm.run/c/c55fe1d6-853b-4113-bdaf-d828bee0da53

Trump: https://chat.vlm.run/c/84b57509-ed38-454a-9b17-85c03a247993

Kohli: https://chat.vlm.run/c/da8ffe8a-0818-4820-bdeb-b82d965ae3a4

Lamar: https://chat.vlm.run/c/ef6488c3-b0f0-4651-944e-7d0c4f98bdd1

Mrbeast: https://chat.vlm.run/c/c24d1cac-feb0-4a53-b1a2-2bbb62a7b7fb

Try the creation engine: https://3d.hunyuanglobal.com/

Access the API: https://www.tencentcloud.com/products/ai3d

https://3d-models.hunyuan.tencent.com/world/worldMirror1_0/H...

Would be great to see video results for this as well. I generated some with other models. Nano pro seems the best so far

No I mean I am an immigrant I am already tracked more than a citizen. We have accepted that fact. But the tracking is done with government portal. But asking third party sources to keep a tab on immigrant can go bad very quickly both for immigrants and citizens. I think it is worse for citizens.

Wow that is crazy may be YC next funded company will be on this. But there are so many ethical considereations here. Tracking immigrant means they will track citizens as well. We need to see how these companies are moderated

I tested it out here https://news.ysimulator.run/item/3196 for ocr, segmentation, detection and 3d in a single chat. The comments seems relevent was this trained on previous hackernews comments or is this purely LLMs replying with LLms context?

great this is more on the techincal details. it is great but would be great to see the data. I know they will not expose such information but would be great to have a visibility onto the datasets and how the data was sourced.

The model looks good for an open source model. I want to see how these models are trained. may be they have a base model from academic datasets and quickly fine-tune with models like nano banana pro or something? That could be the game for such models. But great to see an open source model competing with the big players.

you can download them at https://github.com/facebookresearch/sam3. for 3d https://github.com/facebookresearch/sam-3d-objects

I am confuded what is 4d about this 3d visualization? What is the camera capturing?

Man this was my favourite method before neural networks. people tried integrating it into models but somehow they never came back. Would love to see them in the future

Should we not start with Ramayana and Mahabartham? Although they are religious texts they had a lot of science fiction storylines.

Very funny i just a got a mail saying the review is ready after 5 hours and the website is down. https://paperreview.ai/

China is leading in a lot of recent technologies. need to see how US will be able to compete with them on these technologies. Even AI the opesource models are all coming from china and US is not able to compete

I’ve taken a more opinionated stance on this. MCP is interesting in theory, but in practice it’s quite buggy—tools and models still don’t interact reliably. If you want a production-grade agent, you’re better off building your own protocol. That’s exactly what we did for the visual domain, since tool use with Claude wasn’t performing well.

Paper: https://arxiv.org/abs/2511.14210

I agree i think specifically the world is multi modal. Getting a chat to be truly multi modal .i.e interacting with different data types and text in an unified way is going to be the next big thing. Mainly given how robotics is taking off 3d might be another important aspect to it. At vlm.run we are trying to make this possible how to combine VLM's and LLM's in a seem less way to get the best UI. https://chat.vlm.run/c/3fcd6b33-266f-4796-9d10-cfc152e945b7

The model is great it is able to code up some interesting visual tasks(I guess they have pretty strong tool calling capapbilities). Like orchestrate prompt -> image generate -> Segmentation -> 3D reconstruction. Checkout the results here https://chat.vlm.run/c/3fcd6b33-266f-4796-9d10-cfc152e945b7. Note the model was only used to orchestrate the pipeline, the tasks are done by other models in an agentic framework. They much have improved tool calling framework with all the MCP usage. Gemini 3 was able to orchestrate the same but Claude 4.5 is much faster

Not sure what is going on but seems like meta is lacking behind other startups and other frontier models in this space. They invested most in Meta reality labs in the last decade more than any other company and they come up with such poor rendering while the competitors are making pretty cool real world demos. Meta should stop thinking of these as research projects and actually spend time building real products with proper 3d rendering.