Hacker News Reader

Comment by thewataccount | Hacker News Reader

I might be having a moment - but I can't find any links to a git repo, huggingface, or anything about the models/weights/checkpoints directly from the article.

I just see a zip download that AFAIK also doesn't contain the weights/checkpoints. I find this a bit odd, the contents of the zip (from the gdrive preview) look like they should be in a git repo, and I assume they download the model from somewhere? GDrive usually has rate limits which I'm concerned about.

If anyone from databricks reads this - are there plans to publish this on a git repo somewhere, as well as the weights/checkpoints?

EDIT: Oh I just noticed

> Contact us at hello-dolly@databricks.com if you would like to get access to the trained weights.

This... seems odd for a article titled "Democratizing the magic of ChatGPT with open models"?

View on HN

Full source code is up here now:

https://github.com/databrickslabs/dolly

Sorry it took us a day to get the external repo setup.

Awesome thank you!

Was the Alpaca dataset being licensed as non-commercial only the reason you aren't releasing the weights? Is it possible to just release them under the same license?

Yes the issue is that some of the training data is arguably tainted with some noncommercial license (it's nuanced, discussed below in my comment). We are releasing weights to people who request but we just wanted to have an email request flow so that we can make sure people know it's just for noncommercial purposes.

Working on a model without this issue. Certainly our goal is totally open models anyone can use for anything.

Understandable, thank you for the response!

I've been a bit jaded by the "open/democratizing ai" stuff and then having companies stiff us at actually making it open - but not wanting to be the first to litigate these new types of issues ml brings is very understandable.

Question - Would you consider benchmarking a single 4090 for your training? While training in a few hours with 8x A100's is impressive, myself and I think others are curious how that translates to consumer hardware. IMO running/fine-tuning on consumer hardware is the ultimate endgame for all ai models.

Look forward to a response. We are heading toward a 6X Bizon 4090 system as a test bed.

https://bizon-tech.com/bizon-zx5500.html

The README also says this:

> This fine-tunes the [GPT-J 6B](https://huggingface.co/EleutherAI/gpt-j-6B) model on the [Alpaca](https://huggingface.co/datasets/tatsu-lab/alpaca) dataset using a Databricks notebook.

> Please note that while GPT-J 6B is Apache 2.0 licensed, the Alpaca dataset is licensed under Creative Commons NonCommercial (CC BY-NC 4.0).

...so, this cannot be used for commercial purposes

> ...so, this cannot be used for commercial purposes.

The legal relation between models and training data sets seems murky; of course, with the build tooling, you can also substitute in another instruction-following training set if you want to avoid licensing issues with the Alpaca set, whereas if you aren't concerbed with them, you can just blaze ahead.

> ...so, this cannot be used for commercial purposes

Can't they also release the fine-tuned weights as non-commercial as well?

Essentially every model worth anything has been trained on a unfathomably large amount of data under copyright, with every possible licensing scheme you could imagine, under the assumption that it is fair use. While you can argue that it's all built on a house of cards (and a court may well agree with you) it's kind of arbitrary to draw a line here.

Fair use via "this is completely impossible to regulate so you might as well embrace it"

> under the assumption that it is fair use.

No, because you as a human looking at "art" over your lifetime and learning from it is not "fair use" of the copyright, it's no-use at all. This is the crux of every argument for both for language models and AI Art models, that their tools are learning how to draw, learning what styles and characteristics of input art correspond the most with words, and creating art with that knowledge just like any other human, not simply collaging together different pieces of art.

As far as I know the copyright situation for models is ambiguous and also depends on the region. In the US you can't copyright data made by an automated process but you can in the EU, or something to that effect.

> ...so, this cannot be used for commercial purposes

or you can raise $30,000,000 right now and worry about the copyright infringement lawsuit in 2026 or never.

> ...so, this cannot be used for commercial purposes

The implication being that you're only "democratizing" something if people can make money off of it?

Kinda? I (personally) read "democratizing" as intending to be for the benefit of many over the few. Bit duplicitous to preclude access to the means of production in that definition, "for many rather than the few (BUT, wait, the actual economic benefit and utility is still locked away for the few)".

But maybe "democratize" is starting to mean something similar to "open". All the good words.

Lol. This is classic ML crap. Files with no documentation, no links, multiple files with the same-ish name but no explanation for which one is what.

Yes, the ZIP on Google Drive owned by one of their engineers is weird considering they have a pretty active GitHub presence of open source projects, though it does use an Apache license like their others.

Perhaps Databricks suspected another big announcement coming soon and wanted to get this announcement out?

Are they pulling a Facebook, on model access?

I think they are dodging unclear legal issues surrounding certain steps of the model-building process while being as open as possible with the components given that constraint, allowing downstream users to make their own legal risk vs. effort choices.

Given the hardware/energy need to train it be nice, to have a legal document that said something like this model has no warranty, it may be a break through machine or a hand grenade. Use at you own risk!

Yes, this.

From what I can tell they're fine-tuning EleutherAI's GPT-J.

Alpaca was made to fine-tune LLaMa, however they also released their dataset they used to do this, and it looks like Dolly is this dataset applied to GPT-J, and does not use LLaMa itself.

Thanks I missed that email while skimming

So it’s another classic private only model that they’ll pull as soon as the suckers have trained it up for them

Databricks is on a roll

Reply on news.ycombinator.com