Comment by JonathanFly | Hacker News Reader

Comment by JonathanFly | Hacker News Reader

Parent Full thread

I'll link my Bark fork with long audio generation and other features on the root thread, I suppose: https://github.com/JonathanFly/bark

There's going to be a big update this week with some new stuff I haven't talked about. And a bunch of amazing, clear voices, with a huge variety of styles, that blow the default Suno voices out of the water. Arguably even better than Eleven in some ways. I'm excited even though I have nothing to DO with the voices!

Don't get too attached though. I was just playing around and made a Bark fork and it got more popular than expected. And now I'm dreading a future full of hours of unpaid support and maintenance that I definitely can NOT afford, for a software product I don't even really have a personal use case for. I'm not generating my own audiobooks or anything, I won’t be using it long term myself, I was just curious what Bark could do. (Turns out a LOT more than you might think at first glance, as you'll see this week.) So I'm already trying to work out how I can elegantly wind this thing down and transition people somewhere else. But I'll keep it updated for at least a little while.

Thanks for your fork, I know a lot of folks love it. This might not be the HN style, But I'd suggest you a freemium microSaas approach. A lot of people I know want 1-click solution for audios and you might have great potential for that. That can add some $$$$ to your bank.

Thanks. I'll consider it but I haven't deployed a model or service like that, so it'd be more of a second project in itself than a funding mechanism, probably. I was just realizing this morning how far behind I am on paying work from getting overly distracted by Bark lately. And it's a lot behind so it was kind of wake-up call. Though some of that is adding new features and trying new ideas (some to be seen later on the public fork) not all is support and stuff. Bark is a wild model, a lot of silly ideas kind of work.

I wish there was an easy way to fine tune bark, so we could truly clone our voice for bark inference.

Sadly the bark-voice-clone fork doesn't do it. The voices sound nothing like yourself.

Your gradio gui is great. But I don't understand where to copy the cloned npz files to. Even after refreshing the gradio GUI, the ClonedVoices don't appear in the Speaker or Generated Speaker dropdown.

I barely touched that, it's just from the Serp cloning github, but people kept asking so I put it in. Their clone just isn't really doing much though, it's just loading up the coarse model with the encoded wav file as a fake last generation history to the current segment, and that's basically it. But a lot of the voice is in the semantic model so just doing the coarse doesn't get you much. And even if the semantic model didn't matter as much, the coarse model has both semantic and coarse tokens as inputs and your injected coarse tokens aren't going to line up just right with what a true Bark generated pair of tokens would look like. So what you get is like a robot clone that has the most superficial similarity and lacks the depth of cadence that makes Bark awesome. That's in best case, more often get voices full of static or that don't even read the text you give them. (To be fair, any bark voice can do occasionally not read the text, it's a risk.)

I'm sure somebody will train a model that actually maps an input text to the Bark semantic representation, it shouldn't be that hard, it's just been a few weeks. But the existing clone there is just so primitive I don't know how it got so popular.

> I’m sure somebody will train a model that actually maps an input text to the Bark semantic representation,

But, clearly, that’s already part of the Bark model, so in the abstract one should be able to leverage the existing model with appropriate code to do that, rather than developing a new model. This seems too simple but…isn’t that just “generate_text_semantic” from the existing source (with None as the history_prompt. since you don’t want it in the context of some pre-existing speaker)?

EDIT: Looking at the SERP voice clone, that’s what they are doing. The one thing that I’m intuitively skeptical about (and this is way out of the kind of programming I do normally, so I could be way off) is that the temp they use is kept at the level normally used for synthesis (0.7). I’d think you’d want the temp low, since you’d want generating a baseline for a new speaker to be more deterministic than generating content from an existing speaker.

Leverage sure, but you still basically need a model that does the opposite. You can't like take OpenAI Whisper, which turns speech to text, and just run the model backwards and generate audio. For example. I mean you probably could with some work, but not out of the box.

So, we’re talking about a model that takes an input audio and returns a Bark history prompt, not text -> Bark semantic prompt, right?

If so, yeah, I agree that’s tricky.

If it’s as good as you say, set up a sponsorship goal and we’ll contribute. 11 is massively expensive to use in embedded in an application so anything that’s lightweight enough to be self hosted and can produce 11 labs like output has my dollars.

Reading the comments in your code is hilarious :-)

Thank you.

Reply on news.ycombinator.com