Meta-Transformer: A unified framework for multimodal learning
kxgong.github.io
kxgong.github.io
I think it aims to leverage the cross-modal relationships and unified learning, which might not be possible with expert models designed for only a single modality.
Even if it performs slightly worse on some tasks, the ability to handle multiple modalities within a single framework is an pretty sweet advantage in scenarios where data from various sources need to be processed simultaneously, and patterns across modalities need to be captured somehow.
A general-purpose model could also be a more cost-effective solution in some cases, ensemble experts are difficult to scale and parallelize.
It's an interesting question as it begs questions of conceptual "boundaries."
The sense-plan-do process requires a search and filter process for task switching, assuming an agent can do more than one thing.
So assuming you have a robotic/autonomous agent that is a collection of systems (locomotion, dexterous gripper, visual perception, etc...), if each system could be represented as an "expert module", say for example the dexterous manipulator, then so long as a discriminator can appropriately switch states using the sensor/system inputs, then it's conceptually possible that there is a canonical "expert module" that everyone uses and therefore "general purpose" would apply to the agent as a whole while expert model would apply to the dexterous manipulator.
You can walk that reasoning up the abstraction layers then to conclude that (as usual with these turtle stacks) the distinctions come as each sub system/module specializes more granularly for the environment they operate in.
I think that it's probably forever and always true that any system designed to explore/exploit a bounded environment with comprehensive observations, will always outperform a system that is required to adapt it's sense-plan-do components to the bounded environment without similar observations.
A system would either have to generate different observations than the native agent, or change the boundaries of the environment in a way that is unavailable to the native agent in order to outperform it.
If you had the same quantity of text data as GPT-4 + comparable quantity of data for other domains, it could probably learn transferable skills across those domains.
But it would take a huge amount of processing power that is probably not attainable today
If a general purpose model beats the specialized one, you could almost certainly distill the general purpose one into a better specialized one.
transformer(transformer(transformer( ... x ... ))) = ?
So since it'll be hard to go deeper, going broader by interlacing different model types might be a way to pierce through.
GPT-4 did not scale up substantially in depth, going from 175 b to 220 b per transformer.
Or is the idea to keep the network the same size and trade off some of its nodes for image, video, etc. data?
If so has anyone shown that doing so results in better overall performance?
My lay-observation is that GPT-4 seems to be on the border of usability for most applications so if nothing is gained by simply changing the input data type as opposed to expanding the model then it feels like it won't be of much use yet.
Also apologies if I'm not making sense, I'm almost certainly not using to correct technical terms to articulate what I'm thinking.
Just width if that makes sense. Basically, you add another encoder model but you are not actually increasing the depth that much.
Problem with the assumption though is that transformers are good at identifying and replicating patterns given a set of rules (i.e. how proteins fold and misfold depending on the environment).
Hubble data isn’t so much “we know the rules but not their interactions” as much as “we don’t really know the full set of rules,” so that particular example probably wouldn’t be that fruitful.
In general, biology (where we understand the basic rules but not the complex ways they are combined) is the most fertile ground for transformer driven research.
1. How does the multi-model help here in improving the accuracy of image classification when training is combined from text, images, and audio?
2. How about the speed? I would imagine a model with text, audio and image data would be larger compared to text-only models?
Flying humans was science fiction 120 years ago. A single bomb able to destroy an entire city was science fiction 80 years ago. A machine that can complete more mathematical calculations in one minute than all human manual computation in history was science fiction 60 years ago. EUV photolithography capable of creating molecule-sized transistors was science fiction 30 years ago. A computer that can create visual art and talk to you in plain English was science fiction 2 years ago. A computer that can clone your voice and mannerisms was science fiction 1 year ago.
Science fiction has a way of becoming non-fiction, often within the span of a generation or less.
Nobody's worried about the tech that exists.
> The near term risks we face from AI are constrained to the realms of spam and privacy.
Define "near term".
> World ending super-bots are science fiction.
Science fiction has become science fact before. Where's the knockdown argument that won't happen in this case?
We can deal with the implications of dangerous AI if and when it becomes a problem.
What makes you assume that? We haven't been able yet to deal with the repercussions of globalised social media, we don't even completely understand its impacts. Or dealt with the impact of climate change.
AI seems like a much more encompassing and transformative technology than social media, what makes you assume we will be able to deal with its problems in a timely fashion when they inevitably occur? We might as well not be able to, and as usual, unintended consequences will follow.
> Fear of the unknown shouldn’t be allowed to stop scientific progress.
Scientific progress at any cost while being irresponsible about major consequences of it shouldn't be allowed either, it needs to be a balancing act, just pushing forward without even assessing the risks is a stupid game.
Yes, and they were very worried about igniting the entire atmosphere in a chain reaction when they built the first nuclear bomb, and they only proceeded when they showed that that was very unlikely.
> We can deal with the implications of dangerous AI if and when it becomes a problem.
Can we though? What calculation is this based on?
What calculations is your conjecture that AI will spiral out of control based on? You're not allowed to cite Hollywood movies.
The science behind the nuclear bomb was well understood before any engineering activities began. We have no framework to discuss AGI because it doesn't exist. Your entire premise is based on the idea that we could accidentally create evil AGI without first developing a theory of intelligence. Maybe Newton should have been locked up before he developed his theory of gravity. The man set us on a path toward nuclear weapons, thank god he didn't accidentally build a nuke.
Not will, but a very plausible outcome:
1. An intelligence greater than human intelligence can outthink humans.
2. Artificial intelligence is effectively alien intelligence, and will not innately share human values or thought processes, and could thus be very unpredictable.
3. Artificial intelligence will not have the same physical constraints that humans do (innumerable copies, lack of physical boundaries), and so our ordinary intuitions around containment will not necessarily work.
4. The usefulness of AI means it will be deployed everywhere, controlling and monitoring many things. Combined with the above properties, it will be very difficult to contain, detect, subvert, or eliminate.
5. Millions of billions of AIs will be created, many of which will eventually match or exceed human intelligence. Alignment has to go right every single time to mitigate risk to humans. It has to fail only once.
You know, the completely obvious properties that anyone who knows anything about computers could come up with if they bothered to give this matter some actual thought without the usual arrogant assumption of human superiority and mastery over dumb machines.
> We have no framework to discuss AGI because it doesn't exist.
Which also means we have no framework from which to build safe AIs that don't want to kill us, experiment on us, or exploit us, or ...
> Your entire premise is based on the idea that we could accidentally create evil AGI without first developing a theory of intelligence.
We tamed fire before understanding chemistry. We invented catapults and crossbows before understanding elasticity and kinematics. We domesticated animals and created agriculture before understanding genetics. We created bridges and buildings before understanding civil engineering. We even created computers (Babbage machine) before understanding computation (Turing machines, lambda calculus). There's a long history of inventions preceding understanding.
For all we know we're one simple modification to the transformer architecture away from truly general artificial intelligence. We don't know what we don't know, and all of the anti-doomers are blatantly overconfident about what we don't know.
Also, my conclusions do not even require that that we lack a theory of intelligence. Software bugs will also apply to alignment code. Alignment has to go right every time, per above. I don't think people have a proper appreciation of the long list of hazards here.
What? Did you not see the Netflix documentary on AI for military use? They literally have AI's that can beat fighter pilots in dog fighting.
Just because it isn't walking around having coffee and chatting you up, doesn't mean it isn't already very advanced and deadly.
It’s scifi themed end-times cosplay.
But the post was just saying it seems 'dangerous'. It is already 'dangerous'.
Yes, it will probably become even 'more dangerous'.
I'd disagree that many people agree on common definitions of risk. Some people think autonomous drones that can beat humans in a dogfight is already too far, others are holding out for some paper clip optimizers before getting worried.
You included 'world ending' as the definition of risk, others have lower bar than that.
But the more different technology is plugged together to start resembling a brain, like a visual cortex, a speech center, motor controls, etc...
At some point the distinction between carbon based life and silicon becomes meaningless vanishes. All the arguments or proofs that humans are conscious would equally prove AI is conscious. Or that neither truly are. Proving an AI is not conscious would also prove humans aren't.
And of course, Terminators.