Audiobox: Meta's new foundation research model for audio generation

>If the current pace of progress continues, we'll see games with complete narrative freedom for players, where you aren't limited to pre-written answers anymore, but can actually talk to in-game characters with your actual voice, goals, and motivations. And those virtual conversation participants can talk back to you, react to your words and actions in a believable, fully immersive manner. That's a dream come true for every gamer on the face of this earth, I believe.

It's basically Dungeons and Dragons with an AI dungeon master who can generate video in realtime. Which would be awesome, but like Dungeons and Dragons it wouldn't be easy to keep the player on track.

I’m an amateur music producer and vocals are by far the toughest part of making music. I have to find a singer, convince them to work with me (I am an amateur and not particularly good tbh), and book studio space because its very tough to get a clean recording at home.

I’m hoping that like digital instruments, I’ll be able to splice in digital voices instead of finding singers.

It'll affect linear entertainment far before it impacts games seriously (Especially the 'full immersive' games where you chat with AI agents)

AI is still too expensive and performance intensive to run in games cost effectively, and truly powerful AI is probably another 10-100x cost increase.

On the other hand, novels will be rapidly replaced by visual novels. The cost of having a novel fully illustrated and voiced will go down 1000x. A high quality illustration used to cost $500-$1000 (A day's work from a high-tier commercial artist), soon it will be about $0.5. I'm not counting in the author's time to prompt the images, because it would have costed them way more time to communicate with the illustrator anyways.

The entire boundary between novels, comics, cartoons etc will blur. Like if a newly written Harry Potter can have thousands of illustrations set in Hogwarts and be fully voiced, the standards for a movie adaptation will be astronomically high, which will in turn drive AI use in movie production just to keep up.

> react to your words and actions in a believable, fully immersive manner

“I’m sorry, but as an ethically trained AI I cannot engage in this sword fight. Violence is never the answer.”

Yeah. It’s gonna be very immersive :P

Back to Descartes

Approaching a full life simulation.

Maybe soon game developers will realize black holes and the uncertainty principle are great for efficiency.

> The more rational voices in my mind, though, become more and more afraid of a world where the only thing you can trust is people sitting right in front of you. That makes the world of information pretty small again.

Gives me cosmological analogies: in the far enough future, the only things you can see in the night sky are the members of the local supercluster.

I used to be an LLM until I took an arrow to the knee. Aside from the joke, I think the barrier would definitely reduce and in-game characters would be contextually far more aware but how do you enforce plot progression in such a truly open world? Can you control the boundary of LLM expression?

It would be fantastic to put a bunch of different LLMs in a game map with “senses” fulfilled by multimodal inputs and agency to carry out actions within the game’s universe. With a goal such as make the most money or rule the most kingdoms, it would be super interesting to see how it self organizes.

The amount of context needed would require quite a bit of novel R&D that doesn't exist on the horizon yet. I think it's more realistic that it'll be a mixture of real & fake in the interim (e.g. the LLM will record important game state changes / information & then use that as context but it'll still forget a bunch of things you'd expect a human to).

Part of me looks forward to these experiences. A larger part of me already mourns the decline is purely human creativity that they suggest. Perhaps AI models will make the perfect game or music or write the perfect novel. I'm certain they'll be programmed to reproduce the funky jank and humans bring to everything they create. At some level, though, we each have our own story that we want to tell and when there are so many automated voices in the room, it's going to be harder to tell those stories.

Definitely agree with your last point, but regarding games with endless possibilities. Mm I rather have a game that is created by someone with a clear goalbin mind. One with boundaries. A good single player experience. Like for example Alan wake 2.

I'm old and I sympathize with your last sentence, I _imagine_ certain reinforced paths through repeated experience are myelin covered so much as to be almost static. But if I think of generations growing up, their brains aren't reinforced/seasoned enough to spot these "Wait... What?" moments in online content that is almost sensical but not quite. In a world with ever increasing AI hallucinated content, when children absorb fake content and build strong paths in their brains, at some point the words/meaning could be so distorted that you cannot understand anymore the person in front of you? I think of the political landscape where we have similar problems, social media algorithms curate content to keep you entertained within subjects A and B, and you have a community with shared values and you are seasoned and share its language domain. How would the landscape look like when the net can be bombarded by content that appears to be true, valid and useful but it really is not, for young generations? Also if somebody can paint the right myoline picture in my head (not plausible text generation but science) I would really appreciate it.

> That's a dream come true for every gamer on the face of this earth, I believe.

There's definitely some gamers who would like to never talk to a character in a video game, even if it means hours of bumbling around for the blue key that some NPC just told me about while I ignored 100% of the text in the game.

So on one hand there is the prospect of "complete narrative freedom for players" and on the other hand there is a fake information dystopia right ahead of us. Well, I would suspect, that the more dystopia we are going to have, the more people will want to flee that reality for a while into ever getting better role playing games. Is that, what people once called progress?

"a world where the only thing you can trust is people sitting right in front of you"

Oh and I think real people can be a source of missinformation as well. Also holograms(or brain implants) might have a breakthrough soon, as well. So all in all I think we are living in interesting times.

> The more rational voices in my mind, though, become more and more afraid of a world where the only thing you can trust is people sitting right in front of you.

You could care less if the content you already consume on forums like HN were generated by AGI. Just sayin.

Thanks for the link. License is clear: Researchers and developers are free to use the codes and model weights of both Qwen-Audio and Qwen-Audio-Chat. We also allow their commercial use.

and important, if you have more than 100m active users:) 4. Restrictions If you are commercially using the Materials, and your product or service has more than 100 million monthly active users, You shall request a license from Us

So, looks like it's absolutely fine to use, except for IT behemoth.

As for how to use, API, I think. Interesting applications are possible. Like interactive mobile robots. Assistants for people with disabilities, both software and wearable.

Interesting times... this will be called AI revolution probably. It's already not a joke, after several ups and downs.

> Alibaba's equivalent was released earlier in Nov and it's open-sourced! https://github.com/QwenLM/Qwen-Audio

Openly distributed perhaps, but definitely not open source. The license appears as closed as meta (research-only with some leeway for other uses.)

Do you have any truely open-source general audio generation models yet?

I know about StyleTTS2, which is open source (MIT) and uncensored, but that model focuses on speech generation only. Having an non proprietary model like audiobox or Qwen-Audio would be really nice.

Normally I want basically everything to be open source. But as soon as I saw that audio restyling demo, I began to feel concerned they may be open sourcing this. The model can take a sample speakers voice, new text to speak, and also a description of a new location (like a cathedral with many echoes, or other background noises) and produce a convincing new audio sample.

This technology will present serious challenges for the verification of covertly recorded audio. It will of course ultimately become widespread but I’m not inherently bothered by the idea of slowing down its release. Giving researchers extra time to examine possible detection techniques seems helpful to me.

Have you seen some of the demos people have been building with Unreal 5.3? Insane stuff. In a decade, this stuff will be hard to tell from reality.

This is why I'm high on the metaverse long term. In ten years, there will be a $500 (or whatever the 2033 inflation adjusted value is) VR headset that blows the Apple Vision Pro out of the water in terms of optics, will run a highly optimized version of the lastest revision of Llama locally (and it will be much better than anything we currently have today), come with wifi 8 (so it will have multigigabit per second real word performance), capable of rendering graphics that look much more realistic than Unreal Engine 5 (with high frame rates due to AI upscaling and frame generation).

There will be people that will spend almost every waking hour with one of those things attached to their face if they can also make this device lightweight and comfortable

By the time a machine learning model can replace 1000 workers they’ll just stop hiring real workers. What remains will be tasks which can’t be automated.

What you're looking for is embodiment, and actually this was explicitly left aside in a recent paper that attempts to give measurement criteria for AGI[0]. But I agree with you that the entire lived sensation is critical to approaching any objective involving alignment.

[0] https://arxiv.org/abs/2311.02462

What if the "environment" for an AI is just "the internet".

A long time ago, there was a great story in a Shadowrun supplement of all things about a hacker that got trained to teach an "ai" how to break into computers. It was basically at a child's level, emotionally, and the only world it's ever known was "the matrix" (yes, really -- and written almost a decade before The Matrix came out). Eventually it turns out that it's not an ai, but a corporation was stealing kids and sticking them in Virtual Reality at birth to train a team of super hackers.

I mean sure.. but imagine if this were open sourced as is. This is new tech that has barely had time to mature. The possibilities for abuse are endless. I for one am happy that this model isn't being open sourced. This is an excellent way for people to generate all kinds of disturbing and fake audio clips.

A utopian future would be FOSS LLMs that are private and run locally. The opposite of one where models are public, proprietary - running on your data and owned by just a few large entities.