DALL-E 2 open source implementation
github.com
github.com
https://github.com/lucidrains/DALLE2-pytorch/discussions/10
Inference cost and scale seems to be much more favourable than large language models (for now).
TL;DR: A single NVidia A100 is most likely sufficient; with a lot of optimization and stepwise execution a single 3090 Ti might also be within the realm of possibility.
https://github.com/lucidrains/DALLE2-pytorch/issues/22#issue...
GPT was rumored to cost in the millions, perhaps the hours estimate is conservative?
(Small timer with absolutely no idea of practical ML here)
[1]: https://github.com/alembics/disco-diffusion
[2]: https://twitter.com/midjourney?t=-kKC5UE-gjIkMvAb709SyQ&s=09
The Latent_Diffusion_LAION_400M notebook generates six 512x512 images in about 45 seconds on a K80 on Collab.
DALL-E2 is more complicated but presumably also better optimised.
Certainly one of the creepier TNG episodes even if it was a one off meant to cash in on the popularity of alien abduction in media at the time.
To work out, you'd need a way to keep some test data secret, which in turn kinda kills decentralisation.
1. A researcher puts out a model with bad initial parameters/data.
2. The chain workers/miners train the model as request.
3. The model fails at the test or verification dataset due to the bad setup.
In this case, the miners would not get paid despite doing exactly what was asked of them.
Lack of trust there can be addressed with a few other things like staking, and a broad desire for all miners that people trust the system as a whole. Quite how to design those things is a complex problem but not insurmountable I think.
Disclaimer, this kind of decentralised more useful kind of work is something I'm investigating now.
I am not sure that is a really big turn off.
Not affiliated and haven't researched yet but looks interesting. There are a few other crypto mining companies offering general compute to manage changing power/price/hardware resources.
Can we? I’m not sure about that - one issue being that they’re reproducable, so other people can recreate them without constraints. It turns out to not be that hard.
I think a better explanation is there’s a bunch of spare “AI ethicists” hanging around because they read too much material from effective altruism cultists and decided they’re going to stop the world conquering AGI. But as OpenAI work doesn’t actually produce AGI, the best they can do is make up random things that sound ethical like “don’t put faces in the training output” and do that.
Btw, their GPT3 playground happily produces incredibly offensive/racist material in the same friendly voice InstructGPT uses for everything else, but there they decided to add a “this text looks kind of offensive, sorry” warning instead of not training it in at all. (And to be clear I think that's the right decision, but man some of the text is surprising when you get it.)
But every time I meet someone who talks about this/has related keywords in their twitter profile/is in a Vox explainer, they always have strange ideas about existential risks and how we're going to be enslaved by evil computers or hit by asteroids or reincarnated by future robot torture gods. I think as soon as you decide you're going to be "effective" using objective rational calculations, you're just opening yourself to adversarial attacks where someone tells you a story about an evil Russell's teapot and you assign it infinite badness points so now you need to spend infinite effort on it even though it's silly. You need a meta-rational step where you just use your brain normally, decide it's silly and ignore it without doing math.
Do you have some proof of this? That they follow a thoughtfully considered ethical framework for each project? Because I never see that addressed, at all…
I think the primary reason for the closedness is that it hypes up the research and makes it look more significant than it is. Same reason Palantir is allegedly happy about negative news coverage, it just makes them seem like James Bond villains.
From a skim of the README it seems like install the package, and then find some training dataset, and then ??? to use it.
Most "open source" models don't actully upload the final models either, just the code they used to train them (not sure if this is the case here), so the next step after that is downloading 100G of data and running your workstation for a few days to train the actual model first.
I recall at one point I attempted to run BERT or GPT-J or one of those lighter language models locally, and it was going well until I realized I needed a 24G of vRAM to even load the model hah.
Resting on laurels won't last. If a company stops improving its offer, a competitor might catch up.
Should just be passing `--gpus all` these days, shouldn't it?
No hurdle, it's just that Docker is not part of the standard toolkit of ML people. ML people all use conda (or virtualenv etc.) which already solves most of the dependency problems, making learning Docker not especially appealing.
But virtually all training/inference platform (including the ones used by OpenAI) are using docker. It's not a technical limitation.
If you have a decent amount of VRAM, you can use it to start generating images with their pre-trained models. They're nowhere near as impressive as DALL-E 2, but they're still pretty damn cool. I don't know what the exact memory requirements are, but I've gotten it to run on a 1080 TI with 11gb.
EDIT: I also tried a 980 with 4GB of RAM a while back, but that failed...so you probably need more than that.
I know this wont help you very much. But, either you are willing to spend a lot of time messing around with the code and have access to a good hardware to train the models, or you would have to wait until someone releases something already trained.
It takes a whole team to replicate an advanced model. Take a look at our open source working groups: Eleuther, LAION, BigScience. They worked for months for one release and burned millions of dollars on GPU (gracefully donated by well meaning sponsors).
The BigScience group is massive:
https://bigscience.notion.site/10743770aae24ff3bdc1b938cf454...
Once that's done it might be as simple as: install the package, and then find some training dataset, and then run the training CLI (for days or weeks or more), and then run the image generation CLI. Or even: just download an already trained model and use the image generation CLI immediately.
pip install big-sleep
dream "a pyramid made of basketballs"
Not nearly as cool as the real DALL-e, but maybe I'm missing something.
1. Training is very finicky and time-consuming.
2. PyTorch model files are pretty large.
For point 1, when I was training a different autoencoder for example, I would run it for a week on 4 GPUs, only to return and see that its results are subpar. For point 2, I would get 100 MB model files for a miniscule transformer (relative to DALL-E numbers). Combined with the strong dependence of the transformer on the training data, this can make it problematic when it comes to sharing pre-trained models for tasks that the user doesn't specify a priori.
[1] - https://venturebeat.com/2021/01/05/openai-debuts-dall-e-for-...
Because every one else, too, lacks the time and hardware required to replicate the training steps.
There is just not that many organizations that can pay for 100-200 thousands hours of training on A100 GPUs.
So the reason why the pretained model is not included, is that it does not exist (yet).
That feels more long-term productive to me than crunching coins.
> you can't simply upgrade the software on all the machines
You can update a website. And a website can use the local gpu.
The break the training task in smaller pieces, parallelize and distribute it part is indeed beyond my skills.
How much is this in money?
Pretty expensive all in all
> To train CLIP, you can either use x-clip package, or join the LAION discord, where a lot of replication efforts are already underway.
That's probably why, since OpenAI isn't open source.
They may as well just call it ClosedAI at this point, seems like every interesting thing they do is kept under wraps.
after you get approved on their wait list of course
So what we really need, are ways to train models in a distributed fashion. Their initial goal was to 'democratize AI'. But democratizing these models doesn't just mean giving everyone access to your trained parameters, it also means that everyone should be able to spend an effort on improving it. I think a good solution for this would be huge.
At this point it's more like doublespeak.
Anyway, getting cycles isn't too hard, expect CoreWeave or another compute-centric company to hop on board in due course.
I am :D
How is the replication effort for this project handling that? They have over 600 people on their Discord right now, do they have some way for them to cooperate on training a single model?
Instead they use donated time on a TPU or a GPU cloud provider.
An exception is Leela Zero since you can realistically have it play chess at home.
So instead distribute the funding? And distributing running models too? Something something crowdfunding blockchain cryptocurrency
Or alternatively https://www.microsoft.com/en-us/research/publication/decentr...
hmmm... what about people who mine crypto on their hardware, and then that hardware goes into a pool of shared $ used to rent out a TPU
That said, their commercial APIs for GPT-3, etc. are very good and my monthly bill for using them seems like very good value.
ClosedAI is a bit harsh, but they have definitely both divided the community somewhat on the definition of “Open” and been very comfortable with focusing on their toys and ignoring the conversation. I mean sama, especially when it comes to Worldcoin, has been very direct about ignoring criticism in general.
Years ago I made a shared tensor library[1] which should allow people to do training in a distributed fashion around the world. Even with relatively slow internet connections, training should still make good use of all the compute available because the whole lot runs asynchronously with highly compressed and approximate updates to shared weights.
The end result is that every bit of computation added has some benefits.
Obviously for a real large scale effort, anti-cheat and anti-spam mechanisms would be needed to ensure nodes aren't deliberately sending bad data to hurt the group effort.
There probably won't be demos for a bit, but sooner-than-later.
Any image or video of anything will soon be faked by commoners. This will be a wild ride!
So, back to the topic, we may have a few ways to build new shields against the new weapons, but this will be not trivial in technical and social matters, and their interactions - having the involved parts adequately understand the new realities: "a document may not be just trusted", "there is no boolean trust value but a 0..1 fraction", "signatures may not be a total warranty"...
To some people, "If you received an e-mail using that sender's name it does not mean he sent it; no, easily his account was not hacked (etc.)" is already a massive blow to their inner world, that makes them throw themselves in the chair finding it unbearable if taken seriously and incomprehensible if considered.
And, some people will find it "less painful" to distrust /you/ than to break the laws in their inner world (it's a phenomenon you may also meet in this very forum) - some minds observe a regimen of strict conservative low expenditure. Complexity involves social dangers.
I don't remember ever hearing of "the Socratic principle" – what do you mean by that?
I had a quick google. There are only 12 pages of results for the phrase, i.e. it's not commonly used, and every hit for "the Socratic principle" I find seems different. The first four I found: "Virtue is knowledge", "follow the argument wherever it leads", "Wherever a man posts himself on his own conviction that this is best or on orders from his commander, there, I do believe, he should remain", and "Whenever we must choose between exclusive and exhaustive alternatives which we have come to perceive as, respectively, just and unjust or, more generally, as virtuous and vicious, that very perception of them should decide our choice. Further deliberation would be useless, for none of the non-moral goods we might hope to gain, taken singly or in combination, could compensate us for the loss of a moral good." If I had to say what I thought Socrates' main principle was, it might be "study humans and ethics, not nature/physics or maths". It doesn't sound like you meant any of those, but I'm not sure.
(Socrates was said by the Pythia, the Oracle of Delphi, that he was the wisest man in Athens: Plato has he comment in the Apology that if there is one thing that makes him wise, it could be that he does not delude himself about his ignorance involving all the things he does not know).
To such "Socratic principle on wisdom and knowledge", the wiser you get the more you see limits in your knowledge; to its counterpart the D-K, lack of experience has subjects underestimate their ignorance.
People experienced in dealing with problems experience a large amount of unexpected obstacles, unexpected in the mental framework they brought: after some long while, difficulties and complications are expected preemptively, they are part of the picture one forms. Experience brings diffidence.
i.e. "Experienced people expect obstacles."
The experienced knows that such naïve «mental framework» is very common: he will expect naivety.
Until then, you'll likely be able to use an API to generate images within the next 6 months.
I can think of lots of malicious things it can be used for.
It's become increasingly common to start off articles with an image, even when the article is describing an abstract concept, just to avoid having a "wall of text." If that's what you need to do, it seems like a more interesting alternative to clip art.
If a click generates a deepfake that causes someone to commit suicide, it might be considered lethal in some jurisdiction.
A is a mentally healthy person living a fulfilling life. Will watching a disturbing deepfake affect A severely enough to cause them to commit suicide? No.
B leads a life of extreme isolation and faces troubling financial difficulties. They come across a deepfake that pushes them over the edge at the wrong time and they give up on their life. Is the deepfake to blame? Or were there more important factors that we could’ve focused on to save B and potentially thousands of others.
And since we’re discussing the unintended consequences of deepfakes, let’s not forget that a carefully crafted one can even save lives hypothetically.
Imagine - you could convince a parent that their child has died. Show them photographic evidence. I don’t know how close to the edge they’d need to be.