Meta AI releases CoTracker, a model for tracking any points (pixels) on a video

- It degrades Googles and OpenAis ability to monetize. That alone is worth gold because tech is not a blue ocean anymore - neither from a market opportunity nor from an investor perspective. Your competitors loss is your gain. They can’t use money they don’t make to mount attacks to dislodge customers, investors, mindshare from you. Yes, Meta is weaponizing "open" to hurt their competitors, but we're in a capitalism world, that's about as good as it gets unless you want purity of motives.

- Secondly it commoditizes AI. Zuck believes that his platforms ultimately will benefit if there is more content. Because his platforms sell the ability to show YOUR content over the rest, for coin. Just as with news/mobile games, driving the value of your content lower and lower by fostering the creation/supply of more of it is good for FB and bad for you. You have to advertise to rise above the noise. (See commoditize your complements)

- Thirdly it reinforces the investor narrative that the company has a future because AI. That's important, because by now consensus is that ads future is under pressure of market saturation, regulation, competition (Apple's ability to hurt Meta at will, Amazon's increasing ads monetisation, etc.). There's more than one company that needs that narrative to land, but combined with hurting google, this is pretty good for a company that didn't have a generative AI strategy and was betting on the metaverse a year ago.

- Fourth, it captures open source momentum behind your limited internal efforts. Llama, when it was first released, was not competitive - easily a year behind Google's and more behind OpenAI's efforts - but by being first to capture thousands of capable contributors, just a few months later it is one of the most efficient and scalable LLM systems out there and incorporating many innovations days after their paper releases. Open models alone, without their foundational model, would have tugged along for a while before reaching this state.

What's fascinating about this situation is how it shuffles the big tech positioning. Meta and Apple, purely on a personal leadership bases, are hating each other's guts, but both Apple and Meta need Google/OpenAI to fail, for different reasons

* Meta's products are generally higher tech than people give them credit for. They have a lot invested in having great image and video tech.

* At the scale Meta is at, and at the margins they can operate at, R&D is a valid use funds. Apple has revenue at ~80B/yr. Microsoft have revenue of ~60B/yr. Meta has revenue at ~30B/yr. No one raises an eyebrow when Microsoft announces some go-no-where R&D project.

* I think there might be a Meta bias towards anything with the word "tracking" in it.

My hypothesis is that they want to get the absolute mindshare and adoption and decide the right product/solutions along the way. Note that there are trends (web 1, web 2, mobile etc.) and the leaders tend to grow significantly.

It is a mad rush time with the new generation of AI, and future products are going to be amazing. Meta cannot afford to have just 1 cash cow. Metaverse failed, but they have a real shot here.

A few things I can think of (beyond social networks).

- Enterprise push, starting with very high-end enterprises. Their license structure (free below X million users) allows it nicely. They can afford to have a small team and yet win big. Note that typically 80% of any enterprise products revenue comes from say, 10-20% of top customers.

- Build a platform. Meta learned from Google, and then Apple debacle on how dependent the world is on platform. What would the new OS look like?

- Weaken Google and Apple. Some indications on why they partnered with Microsoft - this also ties to the point above.

Tracking is a core competency for VR headsets. It could also be used for AI-assisted video production, which is the bread and butter of their advertising platform. Users generate videos, and with more powerful editing tools they can churn out more interesting content than TikTok.

Commoditizing your compliment (https://www.joelonsoftware.com/2002/06/12/strategy-letter-v/)

Meta runs on UGC (User Generated Content), so making it easy and cheap to make great content is in their best interests. If they can reduce the cost of making a Billboard 100 quality song to $0, or make it so anyone with a 1.3MP webcam can use AI to get MKBHD level quality streams going, they:

- increase the amount of content being generated

- increase the quality of that content

- make the content cheaper to acquire/license (since there's dramatically more supply)

> Mark Zuckerberg justified the move in a Facebook post: “Open source drives innovation because it enables many more developers to build with new technology. It also improves safety and security because when software is open, more people can scrutinize it to identify and fix potential issues.” [0]

I don't work in the space so I can't comment much, but my team (Server Operating Systems and similar) has been working with Linux, Systemd, Centos, etc for years for essentially the same reason. Working with the community forces you to build better stuff, because, in a healthy community, no one cares that a "Director from Meta wants it" which is the kind of thing that kills proprietary projects in the cradle all the time.

[0] https://www.vox.com/future-perfect/23817060/meta-open-source...

My guess is to 1) stunt the growth potential of their competitors (google, microsoft) before they acquire some sort of generational lead that forces everyone onto their platforms and 2) reset a level playing field for a new generation of startups to someday acquire, that are built on THEIR way of doing things.

Aside from all the points mentioned by the other comments, there's also the factor that Zuck probably finds this kind of research cool, and considering he owns the majority of the votes, he can do almost whatever he wants with Meta.

While 'commoditizing your complements' is one of their strategies. I think it also helps them attract top-tier talent, as who doesn't want the whole world to see and use your cutting-edge developments.

I remember a blog post on HN from an ex-DeepMind/Google Brain person who postulated that the research group existed for visibility; you use the research to attract talent who sees the cool stuff you work on and gets excited, but once hired funnel that talent elsewhere in the company to attract clicks or generate money. Research as advertising.

> Does anyone understand the business angle for meta here with these models?

Pixel tracking is pretty foundational to Augmented Reality: you can calculate objects' position, orientation and speed - it is obvious to me why Meta would do this research, and seed the models to the wider world who can dream up applications for it (in both sense of the word). When you consider other research projects Meta released (image segmentation, digital twins, etc), the AR/VR play becomes a little more obvious.

I can't wait until someone integrates these tracking models with Blender, as a non-expert in object-tracking - this appears to be superior to the tracking pipeline that was available in Blender a few years ago when I looked into it. The fantastic thing about open source is that the project can be used in ways the author never considered.

Industrial history is full of freewheeling corporate research departments inventing the future for the benefit of all stakeholders. And of dangerous examples what happens when you either 1) lack this innovative capacity or 2) lack the foresight or courage to leverage the new technology.

As a ginormous company, given the historical trends, Meta would be rather poorly positioned if it didn't have a research wing.

We can view this also from the point of view how the resources are used for the benefit of innovation. Some companies have a distributed innovation budget (ie. 3M:s 15% time for all). A research department composing 15% of Meta:s personnel costs would be rather huge (I have no numbers what the actual cost is). So having a dedicated research function can be also viewed as a rather cost effective way to stay ahead of the curve if benchmarked in this context.

One way to look at it is it actually prevents competitors from making money.

You release something for free - other companies that could have released it as a product no longer have a business.

It’s no different from the approach of trying to hire all the top tech talent. You deprive your competitors of resource.

Meta has one of the largest supplies of proprietary data in the world (along with Microsoft and Apple). Big improvements in the world's ability to analyze data benefits them tremendously. If they're able to, for example, identify objects in videos and what's happening with them, they can make better inferences about the content of the videos, which lets them make better recommendations, causing people to watch more videos on FB/Instagram and giving them more ad revenue.

Publishing their models doesn't really enable any of their competitors (TikTok is their only real competitor at all), and being a leader in the community and getting top talent to work on these models benefits them.

While they can’t leapfrog Google or OpenAI, they can introduce a nearly unlimited number of competitors and new entrants to the market. All of whom will be aligned to building on this foundation.

Either you claim the ring for yourself or you destroy it.

At the very least, think about how this can be used for features in Instagram. I can't talk about specifics, but there are real applications in existing products.

Can't speak to the overall AI Labs model reasearch but this pixel tracking is directly tied to computer vision and what they are doing at Reality Labs advancing Mixed Reality passthrough with geometry mapping of your surroundings and hand based gesture controls. There is a lot of computer vision baked into those chips and while this is published I am sure they have productized it using more advanced proprietary methods under patent.

Many blue-sky research divisions end up not costing as much as you'd imagine. Patent licensing, the defence value of a big patent portfolio, the occasional 'demo' which turns out to be a huge success and you're best placed to turn into a commercial product in a new field, etc.

It’s a new framework, a new language, and a platform. It’s like open sourcing React, opening Java, or opening the web browser. Controlling the platform has tremendous benefits despite giving it out for free.

I think it is largely driven by LeCun, he has the Bell labs DNA

Watch Caprica (Battlestar Galactica prequel)

Trying to regain some relevance in a world that has largely moved on from caring about it's core product and the added flop of the 'metaverse'. Without doing things like this they won't be able to compete with the large number of AI/ML startups in hiring space either.

It's a big PR bump, can you even remember the last time someone said something positive about that company? For me I think it would be easily more than 10 years ago.

Whilst I generally agree with the consensus in this thread that they're trying to starve competitors of cash, I'm not particularly bullish about Meta's future.

They've sunk an obscene amount of cash into a failed product (the Metaverse), and their core business -- advertising -- is under threat from increased competition and falling engagement with their main platforms. Is Threads winning? I don't think Threads is winning, or that it will.

To me this feels like a big ship's going down, and all the glorious loot is being thrown off to shed weight, and it's floating ashore on our island.

( A "tracking pixel" is a 1x1 remote image embedded in an email (or website) whose main purpose is that loading the URL confirms that the email was opened. )

long experience with tracking.

Demo has OOM error =(

People do seem to vastly under estimate the time span over which Zuckerberg thinks and the end state he is going for. His vision is for a Facebook-like ubiquity of the tech. He sees AI as the pathway there because it means he can remove so many of the hardware sensors he otherwise would need that will be barriers to get the cost low enough. This is how he will make a little bit of money each from literally billions of users the way he has with Facebook. At this stage of development, he just needs these AI problems solved however that happens, whether it's internal or external. Releasing the tech only accelerates his broader version.

I work as a researcher (ML for robotics) inside a product company so I can speak to this a little. The research I work on generally takes a year between conception and publication (three months of tinkering, six months of refinement and benchmarking, three months to write and submit a paper). Now, I work on more than one of these projects at a time but they all have a similar cadence (conference deadlines do that for you), and can be "measured" by management the same way a year-long engineering project is. The questions I'm asked (or ask myself) and the same as when I was a pure engineer. Are you making headway? Are you on schedule? Do you need help? Should this project be abandoned or rescoped if it's not succeeding? Etc.

Is Microsoft a Product company in your eyes? MSR has existed for a pretty long time and they seem to do a mix of self-directed research, publishing papers, etc, and talking to the other teams about the problems they have (eg excel getting FP or programming-by-example features had some MSR involvement. I don’t know how involved MSR were in stuff like some of the novel techniques used in bing or the RDMA-over-Ethernet things Microsoft tried to get to work in their datacentres (don’t recall if private or azure) – those things surely do count somewhat as research even if not MSR).

PTAM was a solution to a very different type of problem. Did you look at the videos? They are almost exclusively tracking points on soft, flexible bodies which is not possible with PTAM.

Which is clearly a good thing for consumers and for startups who would otherwise have no chance of competing with tech behemoths like Google and MS.

This is sadly the standard license for work produced at Facebook AI Research (FAIR) going back at least three or so years. Most researchers are not deeply invested into the details of licensing, so many of them simply assume it is properly open when they see Creative Commons and move on to do more research. This of course ignores the obvious problems with non-commercial licenses, such as the fact that the definition of a commercial entity might very well involve large chunks of the academic world.

Thus, the choice of whatever bloodsu^Wlawyers inside Facebook that pushed this through as the default has had very nasty consequences. However, researchers inside FAIR can apply internally to have an actually open license applied to their work and I have never heard of it being denied. It is not fun (paperwork), but once you sit down and calmly explain the problems with non-commercial licensing and how it likely limits the uptake of their work, many will start pushing for it. My advice is thus just to try to stay positive and talk to the researchers about this rather than try to shame them (not that I have seen this in this thread).

It should also be stated that although this licensing is not "good enough", it is still better than "Open"AI, DeepMind, and many others that gladly just sit on all but a miniscule amount of the code, data, and models they produce.

Fortunately, "Tracking Everything Everywhere All at Once" ([1], [2]) is Apache 2.0 and it's state of art.

1. https://omnimotion.github.io/

2. https://github.com/qianqianwang68/omnimotion

Hmm would this be compatible with GPL ? I think not but not sure.

Anyway this seems like it would be killer for blender to integrate camera tracking.