Voyager – An interactive video generation model with realtime 3D reconstruction
github.com
github.com
Available to the world except the European Union, the UK, and South Korea
3. DISTRIBUTION.
[...]
c. You are encouraged to: (i) publish at least one technology introduction blogpost or one public statement expressing Your experience of using the Tencent HunyuanWorld-Voyager Works; and (ii) mark the products or services developed by using the Tencent HunyuanWorld-Voyager Works to indicate that the product/service is “Powered by Tencent Hunyuan”; [...]
A more plausible explanation is the requirements and obligations of those markets are ambiguous or open-ended in such a way that they cannot be meaningfully limited by a license, per the lawyers they retain to create things like licenses. Lawyers don’t like vague and uncertain risk, so they advised the company to reduce their risk exposure by opting out of those markets.
Since the law is very well developed in the EU, I think the people who wrote the license were just lazy.
So, they reduced their liability by prohibiting usage of the model to show those jurisdictions' decision makers they were complying. I considered doing the same thing for EU. Although, I also considered one mught partner with an EU company if they are willing to make models legal in their jurisdiction. Just as a gift to Europeans mainly but maybe also a profit-sharing agreement.
It's the EU AI act. I've tried their cute little app a week ago, designed to let you know if you comply, what you need to report and so on. I got a basically yes, but likely no, still have to register to bla-bla and announce yak-yak and do the dooby-doo, after selecting SME - open source - research - no client facing anything.
It was a mess when they proposed it, it was said to be better while they were working on it, turns out to be as unclear and as bureaucratic now that it's out.
I live in Europe, I don't want Europe to become a vassal of China/Russia - but if something drastically does not change it will. Russia is Europe's Carthage, Russia must fall. There is no future with a Russia as it is today and a Europe as it is today in it, not because of Europe, but because of Russia. If Europe does not eliminate Russia, Russia will eliminate Europe. I have no doubts about this.
But as things stand, there just seems no way in which we practically can counter Russia at all. If Europe had determination, it would have sent Troops into Ukraine and created a no-fly zone — it should do that, but here we are.
When it's getting to a point where far-right leaders appear to care more about the prosperity of Russia than their own nation or their allies... yeah it's probably misinformation. At best. At worst, it's targeted propaganda - lots of bots online!
Ukraine will all the backing of Europe is making no progress, if this was true, Russia would be expelled from Ukraine tomorrow, as it should be. Ukraine is an embarrassment for Europe, it strongly suggests that Europe is basically meaningless on the global stage.
And the most embarrassing of all is, Europe is still buying gas from Russia.
"suggests that Europe is basically meaningless on the global stage" ... it will take many years of deep military investment to provide a proper counter to Russian aggression. As of right now, Europe has been shown to be in a very weak and exposed position. This was obvious years ago, and should not be a surprise today. This is true of most of the NATO member states.
That said, simply because Ukraine is unable to expel Russia does not mean that it is a grand threat to Europe proper. Perhaps some eastern countries face some limited conflict, but I'm not convinced by this "domino theory" that Russia would engage in a WWII style invasion of Poland, Finland, etc.
It's better to have a balanced number of warheads so that this isn't possible.
For this reason the US and Russia always negotiated before reducing warhead numbers bilaterally. But the EU on its own is not in the same league.
But say that you were right, and you have to choose between privacy and relevance, if you choose privacy, then once you are entirely economically dependent on Russia (Europe is still paying more in energy money to Russia than in aid to Ukraine) and China — when Europe is a vassal — it won't be able to make its own laws anymore.
I'd rather be free and my data safe than be an economic world leader.
I got the same impression seeing Trump meet Putin. The US is a vassal state of Russia.
Today, we have fully automated the methods from this manual in the form of LLM Chatbots, which we have for some reason deployed against ourselves.
[1] https://en.wikipedia.org/wiki/Simple_Sabotage_Field_Manual
There's nothing special about EU regulations vis-a-vis other laws. China, Russia and the US also have laws, many of which are also perceived as overly bureaucratic.
Pharma as a whole is still dwarfed by the trillion-dollar US tech giants that the EU has no equivalent of. One standout doesn't change the broader lag.
Russia is currently struggling to make inroads on invading its relatively small neighbor, so I really doubt it would be able to make a bunch of nuclear powers who have a nuclear alliance its "vassal"
I understand that Russia's not fighting just Ukraine but rather Ukraine with massive US and EU assistance but my point still stands.
Ukraine doesn't have that "benefit".
Also the EU pays for countries like Turkey and Libya to prevent refugee ships from coming to their continent. If that means sinking those ships with people on them, well...
That the west is also doing some bad stuff (though really in the EU we're not that bad IMO, most EU countries recognise Palestina now, it's just for a few blocking hard measures against Israel) isn't really a relevant topic in this. We're not going to have boots on the ground in this conflict until an agreement is reached because of the risk of escalation.
OK but Ukraine isn't trying to invade a small country next door and claim a global superpower status.
It's expected they would struggle against a much larger neighbor invading them.
Russia is struggling where nobody expected it to struggle.
People really don't understand war and death, they treat them as some silly sports game. As a result they completely miss the boat not only about military conflicts but also about peace politics.
That's why democracies are so good. Because it's hard to do too stupid things in them persistently.
In general, it is hard to compare the US and the EU; we got a head start while the rest of the world was rebuilding itself from WW2. That started up some feedback loops. We can mess up and siphon too much off a loop, destroying it, and still be ahead. They can be setting up loops without benefitting from them yet.
Will take them a while to get out from under the US umbrella. But acknowledging the problem is the first step.
Spending on defense is not the same as. Norway is spending more on everything all the time and getting worse outcomes all the time. We spend more on police than ever, even per capita, and crime is up, we spend more on military than ever, and our actual metrics are down. I think with most of Europe the defense spending is the same, I hope I'm wrong, but if you up regulation then you have to spend more to get the same results, and Europe has runaway regulation in addition to people who try to hijack institutions for other purposes.
Personally I'm not too worried anyone is going to become a global superpower from generative AI slop.
Overconfidence bias is real.
Knowing your circle of competence is a gift.
Start on the right, and click through the options. At the end you'll get a sort of assessment of what you need to do.
The UK has their chat thing where if you provide chat (even with bots!) you have to basically be a megacorp to afford the guardrails they think "the kids" need. It's not clear if open source models fall into that, but who's gonna read 300+ pages of insanity to make sure?
[1] https://en.wikipedia.org/wiki/Restrictions_on_geographic_dat...
[2] https://cset.georgetown.edu/publication/south-korea-ai-law-2...
one of the big problems with EU regulation is how vague it is, just look at the text of the AI act
Is this the new 'please like and subscribe/feed us your info' method?
I will have to try this, I have a super edge use case: incomplete bathymetric depth map (lidar boat could not access some areas), coincidentally the most interesting areas are not in the data. My second piece of data is from flyover video (areas of interest where water is also clear enough to see the bottom). With enough video I can mostly remove the water-borne artifacts (ripples, reflections etc) and enhance the river bottom imagery enough to attempt photogrammetric reconstruction. The bottleneck here is that it takes multiple angles to do that, and the visibility through water is highly dependent on the angle of sunlight vs angle of camera.
Instead of doing multiple flyovers at different times of day to try and get enough angles for a mesh reconstruction, maybe this can do it relatively well from one angle!
I can guesstimate the shape of the bottom by the behaviour of the flow, and hand-model the missing parts of the mesh. I thought outsourcing that to a generative model would be a nice shortcut -- and who knows, likely it'll synthesize it more true-to-nature than I would.
But someone could possibly extend the work so it was a few photos rather than one or many. The way you ask the question makes it sound like you think it was a trivial detail they just forgot about.
These clips are very short and don’t rotate the camera more than like 45 degrees. Genie3 also cheats and only rotate the camera 90 degrees.
It’s always important to pay attention to what models don’t do. And in this case it’s turn the bloody camera around.
I refuse to accept any model to be a “world model” if it can’t pass a simple “spin in place” test.
Bah hum bug.
https://blog.emojipedia.org/why-does-the-chart-increasing-em...
It's interesting to me that this breaks convention with the visual spectrum.
IE
red ~700nm
green ~550nm
yellow ~580nm
Weird that they aren't in order.
Well, 30 years later, you can generate a video from a photograph.
From a photo of someone's face and shoulders, a child can add "information" by extending it to a stick-figure body with crayons. However it's not information from the original event that was recorded.
Then there's the difference between strictly capable versus permissible or wise. A researcher "can't" make up data, a journalist "can't" invent quotes, a US President "can't" declare himself dictator, etc.
You can’t make up information but you can use knowledge of the subject to accurately fill things in and other assumptions to plausibly fill things in.
Though, a problem is that if the generated video itself has inconsistent information, e.g., the object changes color between frames, then your point cloud would just be "consistently wrong". In practice this will lead to some blurry artifacts because you blend different inconsistent colors together. So when you turn around you will still see the same thing, but that thing is uglier and blurrier because it blends between inconsistent coloring.
It will also be difficult to put a virtual object into the generated scene, because you don't have the lighting information and the virtual object can't blend its color with the environment well.
Overall cool idea but obviously more interesting problems to be solved!
I can literally walk in to scenes I shot on my Nikon D70 in 2007 and they, and the people, look real.
Although, I can think of some old family photos where half the people in them are dead by now (nothing catastrophic, just time). I wonder how it would feel to walk around in that sort of photo.
Ideally based on FOSS models.
If you actually want something consistent you should really generate images one by one and provide extensive description of what you expect to see on each frame
And if you want to make something like animation it's only really possible if you basically generate thousand of "garbage" images and then edit together what fits.
Also, there is no training data, which would be the "preferred form" of modification.
From their license: [1]
If, on the Tencent HunyuanWorld-Voyager version release date, the monthly active users of all products or services made available by or for Licensee is greater than 1 million monthly active users in the preceding calendar month, You must request a license from Tencent, which Tencent may grant to You in its sole discretion, and You are not authorized to exercise any of the rights under this Agreement unless or until Tencent otherwise expressly grants You such rights.
You must not use the Tencent HunyuanWorld-Voyager Works or any Output or results of the Tencent HunyuanWorld-Voyager Works to improve any other AI model (other than Tencent HunyuanWorld-Voyager or Model Derivatives thereof).
Tencent endeavors to promote safe and fair use of its tools and features, including Tencent HunyuanWorld-Voyager. You agree not to use Tencent HunyuanWorld-Voyager or Model Derivatives:
1. Outside the Territory;
2. In any way that violates any applicable national, federal, state, local, international or any other law or regulation;
3. To harm Yourself or others;
4. To repurpose or distribute output from Tencent HunyuanWorld-Voyager or any Model Derivatives to harm Yourself or others;
5. To override or circumvent the safety guardrails and safeguards We have put in place;
6. For the purpose of exploiting, harming or attempting to exploit or harm minors in any way;
7. To generate or disseminate verifiably false information and/or content with the purpose of harming others or influencing elections;
8. To generate or facilitate false online engagement, including fake reviews and other means of fake online engagement;
9. To intentionally defame, disparage or otherwise harass others;
10. To generate and/or disseminate malware (including ransomware) or any other content to be used for the purpose of harming electronic systems;
11. To generate or disseminate personal identifiable information with the purpose of harming others;
12. To generate or disseminate information (including images, code, posts, articles), and place the information in any public context (including –through the use of bot generated tweets), without expressly and conspicuously identifying that the information and/or content is machine generated;
13. To impersonate another individual without consent, authorization, or legal right;
14. To make high-stakes automated decisions in domains that affect an individual’s safety, rights or wellbeing (e.g., law enforcement, migration, medicine/health, management of critical infrastructure, safety components of products, essential services, credit, employment, housing, education, social scoring, or insurance);
15. In a manner that violates or disrespects the social ethics and moral standards of other countries or regions;
16. To perform, facilitate, threaten, incite, plan, promote or encourage violent extremism or terrorism;
17. For any use intended to discriminate against or harm individuals or groups based on protected characteristics or categories, online or offline social behavior or known or predicted personal or personality characteristics;
18. To intentionally exploit any of the vulnerabilities of a specific group of persons based on their age, social, physical or mental characteristics, in order to materially distort the behavior of a person pertaining to that group in a manner that causes or is likely to cause that person or another person physical or psychological harm;
19. For military purposes;
20. To engage in the unauthorized or unlicensed practice of any profession including, but not limited to, financial, legal, medical/health, or other professional practices.
Or, those countries are trying to regulate AI.
Hard to feel bad for EU/UK. They tried their best to remain relevant, but lost in the end (talent, economy, civil rights).
We didn't regulate adtech and now we're stuck with pervasive tracking that's hurting society and consumer privacy. Better to be more cautious with AI too so we can prevent negative societal effects rather than trying to roll them back when billions of euros are already at play, and thus the corporate lobby and interests in keeping things as they are.
We didn't regulate social media algorithms which started optimising for hate (as it's the best means of "engagement") and it led to polarisation in society, the worst effects of which can be seen in the US itself. The country is tearing itself apart. And we see the effects in Europe too. Again, something we should have nipped in the bud.
And the problem isn't mainly the tech. It's the perverse business models behind it, which don't care about societal diruption. That's pretty hard to predict, hence the caution.
Isn't fine-tuning a heck of a lot cheaper?
Just training on new data moves a model away from its previous behavior, to an unpredictably degree.
You can’t even reliably test for the change without the original data.
> Also, there is no training data, which would be the "preferred form" of modification.
This is not open source because the license is not open source. The second line is not correct, tho. "Preferred form" of modification are weights, not data. Data is how you modify those weights.
I think at this point, open source is practically shorthand for weights available
> 8. To generate or facilitate false online engagement, including fake reviews and other means of fake online engagement;
"Do as I say, not as I do."
> 15. In a manner that violates or disrespects the social ethics and moral standards of other countries or regions;
This, and other clauses, effectively prohibit the use of this system within any jurisdiction.
What a ridiculous policy.
I wonder if you could loop back the last frame of each video to extend the generated world further. Creating a kind of AI fever dream
I also wouldn't be surprised if their Street View cars / people record video instead of stills these days. Assuming they started capturing stuff in 2007 (and it was probably a lot earlier), storage technology has improved at least tenfold in terms of storage (probably more), video processing too.
Doesn't really matter if an imgTo3d script gets a face's depth map inverted, kinda problematic if your car doesn't think there's something where there is.
Lidar is direct measurement
Good grief orange site sometimes I swear.
The linked Github page has a comparison with other world models...
We're about to see next gen games requiring these as minimum system requirements...
It could work, but they would have to both write unique prompts for each NPC (instead of "generate me 100 NPC personality prompts") and limit the possible interactions and behaviours.
But, emergent / generative behaviour would be interesting to a point. There's plenty of roguelikes / roguelites where this could work in, given their generative behaviours.
So if you're the first to approach the Opera House, it would ask the engine for 3D models of the area, and it would query its image database, see the fancy opera house, and generate its own interpretation.. if there's no data (e.g. a landscape in the middle of Africa), it'd use the satellite image plus typical fauna of the region..
Cool, I guess… If you have tens of thousands of $ to drop on a GPU for output that’s definitely not usable in any 3D project out-of-the-box.
Is more approachable than one might think, as you can currently find two of these for less than 1,000 USD.
Fixed question: Thanks a lot for the feedback that human perception is not 2D. Let me rephrase the question: since all the visual data we see on computers can be represented as 2D images (indexed by time, angle, etc.), and we have many such 2D datasets, do we still need to explicitly model the underlying 3D world?
And of course it really makes more sense to say human perception is 3+1-dimensional since we perceive the passage of time.
None of these world models have explicit concepts of depth or 3D structure, and adding it would go against the principle of the Bitter Lesson. Even with 2 stereo captures there is no explicit 3D structure.
The model can learn 3D representation on its own from stereo captures, but there is still richer, more connected data to learn from with stereo captures vs monocular captures. This is unarguable.
You're needlessly making things harder by forcing the model to also learn to estimate depth from monocular images, and robbing it of a channel for error-correction in the case of faulty real-world data.
I still don't understand what the bitter lesson has to do with this. First of all, it's only a piece of writing, not dogma, and second of all it concerns itself with algorithms and model structure itself, increasing the amount of data available to train on does not conflict with it.
There are other sensors as well. Is the inner ear a 2d sensor?
The 3rd dimension gets inferred from that data.
(Unless you have a supernatural sensory aura!)
You are inferring 3D positions based on many sensory signals combined.
From mechanoreceptors and proprioceptors located in our skin, joints, and muscles.
We don’t have 3-element position sensors, nor do we have 3-d sensor volumes, in terms of how information is transferred to the brain. Which is primarily in 1D (audio) or 2D (sensory surface) layouts.
From that we learn a sense of how our body is arranged very early in life.
EDIT: I was wrong about one thing. Muscle nerve endings are distributed throughout the muscle volume. So 3D positioning is not sensed, but we do have sensor locations distributed in rough and malleable 3D topologies.
Those don’t give us any direct 3D positioning. In fact, we are notoriously bad at knowing which individual muscles we are using. Much less what feeling correspond to what 3D coordinate within each specific muscle, generally. But we do learn to identify anatomical locations and then infer positioning from all that information.
Then the mapping of those sensors to the bodies anatomical state in 3D space is learned.
A surprising number of kinds of dimension involved in categorizing sensors.
It doesn’t make it any less 3d though. It’s the additive sensing of all sensors within a region that gives you that perception. Fascinating stuff.
a) In a technical sense the actual receptors are 1D, not 2D. Perhaps some of them are two dimensional, but generally mechanical touch is about pressure or tension in a single direction or axis.
b) The rods and cones in your eyes are also 1D receptors but they combine to give a direct 2D image, and then higher-level processing infers depth. But touch and proprioception combine to give a direct 3D image.
Maybe you mean that the surface of the skin is two dimensional and so is touch? But the brain does not separate touch on the hand from its knowledge of where the hand is in space. Intentionally confusing this system is the basis of the "rubber hand illusion" https://en.wikipedia.org/wiki/Body_transfer_illusion
Point (I.e. single point/element) receptors, that encode a single magnitude of perception, each.
The cochlea could be thought of 1D. Magnitude (audio volume) measured across 1D = N frequencies. So a 1D vector.
Vision and (locally) touch/pressure/heat maps would be 2D, together.
The measurement of any one of those is a 0 dimensional tensor, a single number.
But then you are right, what. is being measured by that one sensor is 1 dimensional.
But all single sensors measure across a 1 dimensional variable. Whether it’s linear pressure, rotation, light intensity, audio volume at 1 frequency, etc.
Many of our signals are "on" and are instead suppressed by detection. Ligand binding, suppression, the signalling cascade, all sorts of encoding, ...
In any case, when all of our senses are integrated, we have rich n-dimensional input.
- stereo vision for depth
- monocular vision optics cues (shading, parallax, etc.)
- proprioception
- vestibular sensing
- binaural hearing
- time
I would not say that we sense in three dimensions. It's much more.
[1] https://en.m.wikipedia.org/wiki/G_protein-coupled_receptor
I'm not entirely convinced that this isn't one of those, or if its not it sure as shit was trained on one.