yeldarb

We'll work on adding this to the repo, but I'll try to summarize here.

Why should I use this? If you're looking to deploy computer vision models to production (on edge devices or self-hosting in the cloud), Roboflow Inference is the easiest way to get up and running quickly & to iteratively improve your model's performance over time.

It's also in very active use with many real-world customers in production. That means we've ironed out (and are continuing to find and fix) many of the bugs and edge-cases that you'd invariably encounter building something on your own or using something less mature.

When should you _not_ use this? For everything else.

* If you're prototyping & hacking on core model primitives there are likely too many layers of abstraction & you should wait to productionize the model until you're happy with the underlying architecture.

* If you're not doing computer vision you're probably better off with something focused on the specific needs of LLMs or other types of models.

* If you need the absolute highest speed possible you may want to use lower-level tools like DeepStream and Triton (for now). Out of the box we sacrifice a bit of speed to get a better interface & end-user experience. We do have a TensorRT provider that sacrifices some of that convenience for speed & are working on additional optimizations (including potentially integrating with DeepStream/Triton behind the scenes).

Edit: @zerojames updated the README.

It’s an easy to use inference server for computer vision models.

The end result is a Docker container that serves a standardized API as a microservice that your application uses to get predictions from computer vision models (though there is also a native Python interface).

It’s backed by a bunch of component pieces:

* a server (so you don’t have to reimplement things like image processing & prediction visualization on every project)

* standardized APIs for computer vision tasks (so switching out the model weights and architecture can be done independently of your application code)

* model architecture implementations (which implement the tensor parsing glue between images & predictions) for supervised models that you've fine-tuned to perform custom tasks

* foundation model implementations (like CLIP & SAM) that tend to chain well with fine-tuned models

* reusable utils to make adding support for new models easier

* a model registry (so your code can be independent from your model weights & you don't have to re-build and re-deploy every time you want to iterate on your model weights)

* data management integrations (so you can collect more images of edge cases to improve your dataset & model the more it sees in the wild)

* ecosystem (there are tens of thousands of fine-tuned models shared by users that you can use off the shelf via Roboflow Universe[1], it plays nicely with our other open source tools like Autodistill[2] and Supervision[3], and there is a ton of content around fine-tuning models you can use with it[4])

Additionally, since it's focused on computer vision, it has specific CV-focused features (like direct camera stream input) and makes some different tradeoffs than other more general ML solutions (namely, optimized for small, fast models that run at the edge & need support for running on many different devices like NVIDIA Jetsons and Raspberry Pis in addition to beefy cloud servers).

[1] https://universe.roboflow.com

[2] https://github.com/autodistill/autodistill

[3] https://github.com/roboflow/supervision

[4] https://github.com/roboflow/notebooks

Thanks for the suggestion! Definitely agree, we’ve seen that work extremely well for Supervision[1] and Autodistill[2], some of our other open source projects.

There’s still a lot of polish like this we need to do; we’ve spent most of our effort cleaning up the code and documentation to prep for open sourcing the repo.

Next step is improving the usability of the pip pathway (the native Python interface was just added; the http server was all we had for internal use). Then we’re going to focus on improving the content and expanding the models it supports.

[1] https://github.com/roboflow/supervision

[2] https://github.com/autodistill/autodistill

Love this. Not because of the Airport Pianos per se, but it reminds me of the Internet's early days where people created websites to just celebrate their niche interests out of love with no profit motive. No ads, no paywalls, no newsletter, no tracking scripts, just wholesome content about airport pianos.

What makes this a “foundation model”?

Roboflow | Multiple Roles | Full-time (Remote, SF, NYC) | https://roboflow.com/careers?ref=whoishiring0823

Roboflow is the fastest way to use computer vision in production. We help developers give their software the sense of sight. Our end-to-end platform[1] provides tooling for image collection, annotation, dataset exploration and curation, training, and deployment.

Over 250k engineers (including engineers from 2/3 Fortune 100 companies) build with Roboflow. We now host the largest collection of open source computer vision datasets and pre-trained models[2]. We are pushing forward the CV ecosystem with open source projects like Autodistill[3] and Supervision[4]. And we've built one of the most comprehensive resources for software engineers to learn to use computer vision with our popular blog[5] and YouTube channel[6].

We have several openings available, but are primarily looking for strong technical generalists who want to help us democratize computer vision and like to wear many hats and have an outsized impact. Our engineering culture is built on a foundation of autonomy & we don't consider an engineer fully ramped until they can "choose their own loss function". At Roboflow, engineers aren't just responsible for building things but also for helping figure out what we should build next. We're builders & problem solvers; not just coders. (For this reason we also especially love hiring past and future founders.)

We're currently hiring full-stack engineers for our ML and web platform teams, a web developer to bridge our product and marketing teams, several technical roles on the sales & field engineering teams, and our first applied machine learning researcher to help push forward the state of the art in computer vision.

[1]: https://roboflow.com/?ref=whoishiring0823

[2]: https://roboflow.com/universe?ref=whoishiring0823

[3]: https://github.com/autodistill/autodistill

[4]: https://github.com/roboflow/supervision

[5]: https://blog.roboflow.com/?ref=whoishiring0823

[6]: https://www.youtube.com/@Roboflow

> Their SKILL tool involves a set of algorithms that make the process go much faster, they said, because the agents learn at the same time in parallel. Their research showed if 102 agents each learn one task and then share, the amount of time needed is reduced by a factor of 101.5 after accounting for the necessary communications and knowledge consolidation among agents.

This is a really interesting idea. It's like the reverse of knowledge distillation (which I've been thinking about a lot[1]) where you have one giant model that knows a lot about a lot & you use that model to train smaller, faster models that know a lot about a little.

Instead, if you could train a bunch of models that know a lot about a little (which is less computationally intensive because the problem space is so confined) and combine them into a generalized model, that'd be hugely beneficial.

Unfortunately, after a bit of digging into the paper & Github repo[2], this doesn't seem to be what's happening here.

> The code will learn 102 small and separte [sic] heads(either a linear head or a linear head with a task bias) for each tasks respectively in order. This step can be parallized [sic] on multiple GPUS with one task per GPU. The heads will be saved in the weight folder. After that, the code will learn a task mapper(Either using GMMC or Mahalanobis) to distinguish image task-wisely. Then, all images will be evaluated in the same time without a task label.

So the knowledge isn't being combined (and the agents aren't learning from each other) into a generalized model. They're training a bunch of independent fine-tuned models for specific tasks & adding a model-selection step that maps an image to the most relevant "expert". My guess is you could do the same thing using CLIP vectors as the routing method to supervised models trained on specific datasets (we found that datasets largely live in distinct regions of CLIP-space[3]).

[1] https://github.com/autodistill/autodistill

[2] https://github.com/gyhandy/Shared-Knowledge-Lifelong-Learnin...

[3] https://www.rf100.org

The image functionality is pretty underwhelming.

Some things I tried that it failed at:

* Simple "spot the difference" a young child could do

* Counting chess pieces on a board or coins in a mario screenshot

* Describing a photo of a car on a road

* Evaluating a graph of y=x and one of y=x^2

* Explaining a meme image (said it can't do anything with images of people)

* Taking a screenshot & giving HTML for it (gave random HTML unrelated to image)

* Taking an image and converting to SVG (gave random SVG unrelated to image)

* Describing a photo of a car (it was parked on a road, Bard said it was driving in front of a brick wall)

It was quite good at OCR though even for images that are pretty tough for other models (eg serial numbers on industrial parts).

Sad, I had a lot of fun with it making StackRoboflow[1] (This Question Does Not Exist) a few years ago.

The models (AWD-LSTM and GPT-2) weren't good enough back then to usefully answer programming questions -- but it's super cool to see that vision realized with GPT-4 and other modern LLMs.

[1] https://stackroboflow.com

The worst would be a .js TLD; everyone looking for web frameworks like `angular.js` would end up on a website (that probably is not affiliated with that tool) instead of a search result page. Would make it incredibly easy to trick people into installing malicious dependencies.

How does this compare with the data you get from Airdata?

When I was working on an aerial georeferencing project[1] I found that the on-device flight logs didn’t contain all the info in the serverside logs (eg heading, gimbal orientation, GPS coords, altitude)

[1] https://github.com/roboflow/dji-aerial-georeferencing

Channeling my inner patio11 here: charge more! I know it seems like low price is a good value prop but being so ridiculously inexpensive signals low value.

I noticed when running an `npm audit fix` that request has an open vulnerability is deprecated and not accepting security fixes anymore. While we don't use it directly, several of the packages in our dependency chain do. Looks like it still gets over 18M installs per week.

This post spends a lot of time criticizing the "pointless line breaks" but not a lot of time on why this is a problem or "must be stopped". Block or mute the people that annoy you and move on; problem solved.

Meta AI is on fire recently. Segment Anything, Animated Drawings, and now DINOv2... all in a single month!

Meta AI is on fire recently. Segment Anything, Animated Drawings, and now DINOv2... all in a single month!

Not sure if this is what you mean, but I grabbed some of those images & dropped them in to see what it predicted: https://imgur.com/a/CXLmYXo

This is the core of what we do! Previously our job was distilling human knowledge into a model; now that knowledge is starting to come from bigger models with humans managing the objectives vs doing the labor.

> also are you worried about business model if people can distil to run on their own devices (so they dont need to pay you anymore)?

This is probably a risk to the current business model over the long term, but we're constantly working on reinventing ourselves & finding new ways to provide value. If we don't adapt to the changing world we deserve to go out of business someday. I'd much rather help build the thing that makes us obsolete than sit idly by while someone else builds it.

I think of this risk similarly to the way that I'm marginally worried that, in the long run, AGI will obviate the need for my job. Probably true, but the opportunities it will present are far greater & it's better to focus on how to be valuable in the future than cling to how I provide value today.

The feature they discussed re using Segment Anything to greatly speed up labeling is now live in Roboflow Annotate for anyone to try: https://blog.roboflow.com/label-data-segment-anything-model-...

Distilling these big, slow vision transformer models into something that can be used in realtime on the edge is going to be huge.

Depending on what you need in particular for this project you may be able to find some here: https://universe.roboflow.com/search?q=Dolphin

Btw, if you’re doing this for humanitarian/non-profit purposes we’d be happy to consider sponsoring it!

Is there a good heuristic for picking a reasonable number of clusters automatically for an arbitrary set of vectors?

Wow, this is pretty epic. I put it through its paces on a pretty wide variety of images that have tripped up recent zero-shot models[1] and am thoroughly impressed.

We have a similar "Smart Polygon" tool[2] built into Roboflow but this is next level. Having the model running in the browser makes it so much more fun to use. Stoked it's open source; we're going to work on adding it to our annotation tool ASAP.

[1] Some examples from Open Flamingo last week https://news.ycombinator.com/item?id=35348500

[2] https://blog.roboflow.com/automated-polygon-labeling-compute...

Good timing -- we actually just published a tutorial showing how to build semantic image search (including generating vectors with CLIP) with faiss this morning: https://blog.roboflow.com/clip-semantic-search/

We did YC in S20 (fully remote during COVID) and, as a Silicon Valley outsider, it was absolutely, 100%, no-questions-asked worth it for the fundraising credibility alone.

After demo day, we were able to get in the room with dozens of top-tier investors, got multiple term sheets, and were able to pick the seed investors that were right for our business. This is a luxury none of my non-YC founder friends have had (being outside of Silicon Valley it was just such a stark, stark contrast in our fundraising experience to my friends'). Fundraising has always been a struggle and giant distraction for them.

The doors YC opened for us let us focus on building and have had an outsized impact on our trajectory. Re the 7%: I'd much rather have 93% of a giant pie over 100% of a small one.

Edit: I should caveat that YC and VC in general are built around finding & amplifying outliers. If you don't think you are an outlier founder and aren't trying to build an outlier company, our experience probably isn't relevant.

I always like to try these zero-shot models on things outside of the "normal" COCO classes. Here are some chess board queries:

Counting: https://imgur.com/KTuQ1Bv

Parse the chess board: https://imgur.com/2zYFK1P

(Result): https://imgur.com/Ei4MAl7

Few-Shot Object Detection (Pascal VOC): https://imgur.com/gZkDMn8

Few-Shot Object Detection (simplified): https://imgur.com/Hk8QGMd

Not quite there yet. I've been more impressed with the other new zero-shot multimodal models like Grounding DINO and Azure Dense Captioning. Really looking forward to putting multimodal GPT-4 through its paces as well.

If GPT-4 is talking to another instance of itself vs 3.5 are the results similar? Or is it only good at fooling a less capable version?

> But most of the engineers I personally know are sort of squinting at it and thinking, “Is this another crypto?” Even the devs at Sourcegraph are skeptical. I mean, what engineer isn’t. Being skeptical is a survival skill.

> Remember I told you how my Amazon shares would have been worth $130 million USD today if I hadn’t been such a skeptic about how big Amazon was going to get, and unloaded them all back in 2004-ish.

If you buy into the hype & think this wave is going to be equivalently (or 10x) as impactful as cloud computing, what's the equivalent to buying Amazon stock in 2003?

This is really neat. Is the idea that people will finetune from these pre-trained weights?

Would be really curious to learn the delta in model performance on downstream tasks relative to training from randomly initialized weights or generic weights like COCO.

We released a benchmark dataset of real-world tasks shared by users recently that has several aerial detection datasets you might be able to use to measure this: https://www.rf100.org

(And if you need help this sounds like exactly the type of thing we’d be interested in collaborating on!)

Edit: we also have a bunch more aerial datasets listed here that could be useful for this: https://universe.roboflow.com/browse/aerial

The GPT-4 paper & post[1] describe that the original model is pretty good at predicting the probability of its own correctness (well-calibrated confidence) but the post-processing degrades this property:

> GPT-4 can also be confidently wrong in its predictions, not taking care to double-check work when it’s likely to make a mistake. Interestingly, the base pre-trained model is highly calibrated (its predicted confidence in an answer generally matches the probability of being correct). However, through our current post-training process, the calibration is reduced.

> Left: Calibration plot of the pre-trained GPT-4 model on an MMLU subset. The model’s confidence in its prediction closely matches the probability of being correct. The dotted diagonal line represents perfect calibration. Right: Calibration plot of post-trained PPO GPT-4 model on the same MMLU subset. Our current process hurts the calibration quite a bit.

[1] https://openai.com/research/gpt-4#:~:text=GPT%2D4%20can%20al...

Love it; one of the biggest things we've preached to users of our computer vision dev-tool is: it doesn't matter how much time you spend on your model if your data is bad.

When you're trying to get a first version of your CV project launched, you can usually pretty much ignore the model (and pick one up off the shelf) because the time you spend on improving your data is orders of magnitude higher leverage.

Someday the model might become the limiting re-agent but it's much, much later than people assume (if at all). But the model is the "fun" part for technical people to play with (and academia encourages treating your data as an immutable constant) so it gets a lot of attention at the expense of the data.