FastVLM: Efficient vision encoding for vision language models

I grew up in the 80s as a 100% blind child. Technology was by far not as advanced as today. Computers were just coming up when I was around 12. I learnt to type on a oldschool typewriter, and I also learnt to write braille with a pretty heavy full-metal embossing device. OCR was still quite bad. When I switched to what you call high scooll, I used a laptop with integrated Braille display to follow classes. Used good old DOS as OS and Word 5.5 as my "notepad". Except for PC Lingua for Latin, I basically had no tools specialized for learning. A electronic notepad and my brain was all I had to follow school. And I still made it. I have a great job I love, my own appartment, a sweet girlfriend and I am basically completely independent. To a point where I had to forcefully send away my mother since her continued attempts to "help" me were basically detrimental to my own development. I can not emphasis how important it is how you deal with it as a parent. Since parents are indeed the biggest hinderence to development, we have a saying around here amongst disabled people: "additional disability due to parental overprotection" (Zusatzbehinderung Eltern). Please take a moment to understand what this means, without feeling personally attacked. Its important. Your child can leave home around 18, just like every other kid. I did. Don't slow that process down artificially. The more this is prolonged, the harder it gets for the individual to actually obtain independence.

I am telling you this because I read between the lines that you believe current technology is a reason for you to be hopeful. Sure, it should be. But never forget, your child can do much more then you as a sighted person will ever be able to understand. Don't let them drown in your own misery. Let them discover what they can do. You will be surprised what they come up with. And dont fall for Gear Acquision Syndrome. Sure, tools are nice, and they do get better, which is also nice. I LOVE vision models, to stay on topic somehow. However, I still leave my house with only a cane and my phone in my pocket. I do occasionally ask Siri "Where am I" to get an address if I happen to have forgotten where I am exactly, currently. But at the end of the day, my cane is what shows me the way. Most tech is hype, plain old hearing and your sense of touch gets you much farther then you might think.

Wish you all the best for your own journey, and the development of your child.

I think that there is fantastic potential in having open-weight, OS-standard foundation models.

Especially if the API gives opportunity for app developers to load their custom LoRa fine-tunings onto OS-standard foundation models at runtime, then you can (ideally) have the best of both worlds -- fine-tuned app-specific models with reasonable app sizes.

You could probably get away with f16 or even quantize to int8 and have a much smaller model, but your point stands. Users won't be thrilled to download a 500MB model for an app either.

That’s what they suggested about LLMs at last year’s WWDC iirc. The core models are provided by the OS, while apps bring LORAs to fine-tune them / bring custom heads for them.

My guess is that they won’t confirm it unless it’s a big presentation. WWDC maybe?

Do you have a write up of the tech stack and setup? Or willing to give the gist here?

I'd like to make a private Qwen or similar for my kids to prompt with a button and voice control. It doesn't need vision... Although eventually that'd be very cool.

Siri just sucks.

We might not be there yet...

SmolVLM is from huggingface team

cool to see people doing stuff with smaller models

https://huggingface.co/blog/smolvlm

https://arxiv.org/abs/2504.05299

It might be useful for telling Cream of Chicken from Cream of Mushroom, but for locomotion I can't see this adding anything over existing strategies people use to get around sans sight.

"There's a tree. There's a tree. There's a tree. There's a number of pedestrians. There's a tree. There's a sign." does not strike me as useful feedback for getting around.

That and the ability to hold on to knowledge.

I used molmo to parse screenshots in order to detect locations of UI elements. See the repo below. I think Omni parser from Microsoft would also work well.

https://github.com/logankeenan/george

https://github.com/microsoft/OmniParser

You might be interested in https://github.com/OpenAdaptAI/OpenAdapt

What exactly the strategy is?

What strategy is that?

Average delusional hner who really doesn't know what they are talking about

It's for interpreting images, not generating them.

If all you want is OCR, possibly.

I thought they turned sycophancy off...