Show HN: I stripped DALL·E Mini to its bare essentials and converted it to Torch
github.com
github.com
It was originally a fractal image generation app but it's expanded over time and now has a fairly foolproof installer for all the models you're likely to have heard of (those that have been released anyway).
Needs a google sub to run colab (DALL-E itself needs Colab Pro, but other models run on free version).
edit: not local! but very handy.
<caveat emptor>
bare minimum GPU: NVIDIA 2080 with 8GB VRAM
300 Gb of disk space
</caveat emptor>
Its great if you want to run more "classic" AI algorithms as well!
For example, kpop idols. There are high rest 60fps fancams of them dancing, there are high res broadcasts, images from all angles.
There's already a huge market for deep fake kpop and I believe that they are in for a reckoning.
which is an awful awful thing ;)
BTW, basically every “female” along with male out there has high resolution photos on social media.
and this is my fault how?
Lets be real though, it's the anime-style generation models that are going to be the pioneers in this field.
EDIT: as I am being accused of inventing it I will quote the terms of agreement and license, since maybe its own founder seems to not have read it or someone without training on how to write proper terms and agreements made it for them and the restrictive usage of "Material" does apply to its hosted software.
Note that there is no formal definition of "Materials" or "Service", so that it applies to all the contents of the webpage including the software stored there: https://wandb.ai/site/terms
I quote it:
2. Use License Whether you are accessing the Services for personal, non-commercial transitory viewing only (our free license for individuals only), for academic use, or for commercial purposes (our subscription package for businesses), permission is granted to temporarily download one copy of the information or software (the “Materials”) from our website. This is the grant of a license, not a transfer of title, and under this license, you may not: a. Modify or copy the Materials; b. Use the Materials for any commercial purpose, or for any public display (commercial or non-commercial); c. Attempt to decompile or reverse engineer any software contained in the Materials; d. Remove any copyright or other proprietary notations from the Materials; or e. Transfer the Materials to another person or "mirror" the Materials on any other server. This license shall automatically terminate if you violate any of these restrictions and may be terminated by us at any time. Upon terminating your viewing of these materials or upon the termination of this license, you must destroy any downloaded materials in your possession whether in electronic or printed format. f. Utilize our personal license for individuals for commercial purposes and any such use of our personal license for commercial purposes (e.g. using your corporate email) may result in immediate termination of your license.
Why do you think that?
EDIT: I'll edit respond, since you did. Look at sections 3b and 3c in the terms, they cover Models and other user content specifically. Those are user property, not our property. But I can see how this is confusing. We will clarify it.
I quote it:
2. Use License Whether you are accessing the Services for personal, non-commercial transitory viewing only (our free license for individuals only), for academic use, or for commercial purposes (our subscription package for businesses), permission is granted to temporarily download one copy of the information or software (the “Materials”) from our website. This is the grant of a license, not a transfer of title, and under this license, you may not: a. Modify or copy the Materials; b. Use the Materials for any commercial purpose, or for any public display (commercial or non-commercial); c. Attempt to decompile or reverse engineer any software contained in the Materials; d. Remove any copyright or other proprietary notations from the Materials; or e. Transfer the Materials to another person or "mirror" the Materials on any other server. This license shall automatically terminate if you violate any of these restrictions and may be terminated by us at any time. Upon terminating your viewing of these materials or upon the termination of this license, you must destroy any downloaded materials in your possession whether in electronic or printed format. f. Utilize our personal license for individuals for commercial purposes and any such use of our personal license for commercial purposes (e.g. using your corporate email) may result in immediate termination of your license.
I am certainly not a lawyer, however, 3b and 3c (from the terms link you posted) state that user content, including specifically Models, are property of the user.
Are you saying you think there is a conflict between 2 and 3b, 3c, or did you miss section 3?
``` 3. Intellectual Property & Subscriber Content
a. All right, title, and interest in and to the Services, the Platform, the Usage Data, the Aggregate Data, and the Customizations, including all modifications, improvements, adaptations, enhancements, or translations made thereto, and all proprietary rights therein, will be and remain the sole and exclusive property of us and our licensors.
b. All right, title, and interest in and to the Subscriber Content, including all modifications, improvements, adaptations, enhancements, or translations made thereto, and all proprietary rights therein, will be and remain Subscriber’s sole and exclusive property, other than rights granted to us to enable (i) Subscriber to process its data on the Platform, and (ii) us to aggregate and anonymize Subscriber Content solely to improve Subscriber's user experience.
c. Subscriber Content means any data, media, and other materials that Subscriber and its Authorized Users submit to the Platform pursuant to this Agreement, including, without limitation, all Models and Projects, and any and all reproductions, visualizations, analyses, automations, scales, and other reports output by the Platform based on such Models and Projects. ```
This is an ambiguation and it is an issue that as founder you should address as it could be interpreted as a self contradictory agreement and then invalidate part of the agreement (as I've seen with non formal open source licenses, ill formed patents which became bypassed, etc).
A way to solve that might be using a glossary of what do you mean by each term, however I do recommend you to use a lawyer for such things as this sort of mistakes might become expensive later on.
I get this error detokenizing image Traceback (most recent call last): File "/home/ubuntu/work/min-dalle/image_from_text.py", line 44, in <module> image = generate_image_from_text( File "/home/ubuntu/work/min-dalle/min_dalle/generate_image.py", line 74, in generate_image_from_text image = detokenize_torch(image_tokens) File "/home/ubuntu/work/min-dalle/min_dalle/min_dalle_torch.py", line 107, in detokenize_torch params = load_vqgan_torch_params(model_path) File "/home/ubuntu/work/min-dalle/min_dalle/load_params.py", line 11, in load_vqgan_torch_params params: Dict[str, numpy.ndarray] = serialization.msgpack_restore(f.read()) File "/usr/local/lib/python3.10/dist-packages/flax/serialization.py", line 350, in msgpack_restore state_dict = msgpack.unpackb( File "msgpack/_unpacker.pyx", line 201, in msgpack._cmsgpack.unpackb msgpack.exceptions.ExtraData: unpack(b) received extra data.
https://github.com/kuprel/min-dalle/issues/1#issuecomment-11...
What toolchain will get it working on Mac?
~ which gcc
/usr/bin/gcc
Apple clang version 13.1.6 (clang-1316.0.21.2.5)
Target: arm64-apple-darwin21.5.0
Thread model: posix
InstalledDir: /Library/Developer/CommandLineTools/usr/binI couldn't get the "mega" option to work, I got an error "TypeError: lax.dynamic_update_slice requires arguments to have the same dtypes, got float32, float16" (looks like a known issue https://github.com/kuprel/min-dalle/issues/2)
Edit: installing flax 0.4.2 fixes this issue, thank all!
Macbook Pro M1 Pro numbers (CPU):
python3 image_from_text.py --text='court sketch of godzilla on trial' --mega 640.24s user 179.30s system 544% cpu 2:30.39 totalpython3 image_from_text.py --text='a comfy chair that looks like an avocado' 612.30s user 180.72s system 552% cpu 2:23.52 total
https://github.com/kuprel/min-dalle/commit/38ebe54a382f36dc7...
So much for semver :(
> Major version zero (0.y.z) is for initial development. Anything MAY change at any time. The public API SHOULD NOT be considered stable.
Variant schemes like the one respected by Cargo (https://doc.rust-lang.org/cargo/reference/semver.html) aren't usually much different.
> Initial development releases starting with "0.y.z" can treat changes in "y" as a major release, and "z" as a minor release.
It's not listed in the requirements
I've posted it as an issue
Weight & Biases
> Why does it keep asking for an API key
From the README:
the Weight & Biases python package is used to download the DALL·E Mini and DALL·E Mega transformer models
It might not be obvious you need an account if you aren't in the field though.
I'd prefer to download it myself and choose where I put it too.
It now uses some hashed filename in some config directory in your homedir for this, I dislike this and want control over where I put models, make it more self contained instead of random directories spread all over your OS, and give them as input by file path.
This feedback is about dalle mini playground instead but it does the same thing. If this one is stripped to bare essentials I'd expect this type of dependencies stripped too.
Edit: I don't want to seem like complaining too much though and am very happy with these open models and tooling for them. Thanks!
There are tools to convert to intermediate formats, like ONNX, but they are limited and don't work all the time. The automatic conversion tools usually assume that you can trace execution of the model for a dummy input and usually only work well if there isn't any complex logic (e.g. conditions can be problematic). Some operations aren't supported well, etc.
This isn't always technically difficult, but it's tedious because it usually involves double checking that at all steps, the model produces identical outputs for a given input. An additional challenge when transferring weights is that models are fragile and minor differences might have large effects on the predictions (even though if you trained from scratch, you might get similar results).
Also for deployment, the less cruft in the repository the better. A lot of research repositories end up pulling in all kinds of crazy dependencies to do evaluation, multiple big frameworks etc.
Is it a problem of accumulation of floating-point errors in operations that are done in a different order and with different kinds of arithmetic optimisations (so that they would be identical if they used un-optimised symbolic operations), or is there something else in the implementation of a neural network that I'm missing?
Arithmetic differences do happen equivalent ops, but I've not found that to be a significant issue. I was converting a UNet and the difference in outputs for a random input was at most O(1e-4) which was fine for what we were doing. It's more tedious than anything else. Occasionally you'll run into something that seems like it should be a find+replace, but it doesn't work because some operation doesn't exist, or some operation doesn't work quite the same way.
I'm not too familiar with Tensorflow, so I can't give an example there, but a similar issue I recently faced when converting a model from Pytorch to ONNX is that Pytorch has a builtin discrete fourier transform (DFT) operation, while ONNX doesn't (yet. They're adding it). So I had to express a DFT in terms of other ONNX primitives, which took time.
Lambdas and other custom code are also problematic, as their code isn't necessarily stored within the graph.
Graph to graph conversion can be tricky due to subtle differences in implementation (even between different versions of the same framework), but it's perfectly possible, though not many utilities go all the way to graph rewriting if required.
Problems arise with custom layers and lambdas, which are not always serialised with the graph depending on the format.
Not to discourage the OP of course, great work.
On my desktop, running the example
> python image_from_text.py --text='alien life' --seed=7
results in
> RuntimeError: This version of jaxlib was built using AVX instructions, which your CPU and/or operating system do not support. You may be able work around this issue by building jaxlib from source.
Unfortunately, following the instructions to build JAXlib from source (https://jax.readthedocs.io/en/latest/developer.html#building...) result in several 404 not found errors, which later cause the build to stop when it tries to do something with the non-existent files.
Unfortunately, it looks like I won't be running this today.
python3 image_from_text.py --text='a happy giraffe eating the world' --seed=7 154.61s user 22.18s system 262% cpu 1:07.40 total
WARNING:absl:No GPU/TPU found, falling back to CPU. (Set TF_CPP_MIN_LOG_LEVEL=0 and rerun for more info.)
I assume it would be much faster with a grunty graphics card
Mini = 5.33 s
Mega = 14.7 s
Update: about 1/2 that time is just loading the model, so if you load the model and then generate multiple images, it drops to:
Mini = 3.91 s
Mega = 8.86 s
> wandb artifact get --root=./pretrained/dalle_bart_mini dalle-mini/dalle-mini/mini-1:v0
and select option (3)
EDIT: wandb artifact get --root=./pretrained/dalle_bart_mega dalle-mini/dalle-mini/mega-1-fp16:v14
Note, it is a 4938MB download
If it depends on the hardware, what would be the limit when one rents the biggest machine available in the cloud?
You can technically increase or decrease this - or use a different aspect ratio by using more or fewer image tokens, but this is static after you start training. It will also require more "decodes" from the backbone VQGAN model (responsible for converting pixels to image tokens), and thus take longer to run inference on.
CLIP-guided VQGAN can get around this by taking the average CLIP score over multiple "cutouts" of the whole image allowing for a broad range of resolutions and aspect ratios.
Edit: actually it's easier to open a terminal and move /content/pretrained/vqgan to /content/min-dalle/pretrained/vqgan
UnfilteredStackTrace Traceback (most recent call last) <ipython-input-2-0e20e3adf861> in <module>() 2 ----> 3 image = generate_image_from_text("alien life", seed=7) 4 display(image)
67 frames UnfilteredStackTrace: TypeError: lax.dynamic_update_slice requires arguments to have the same dtypes, got float16, float32.
The stack trace below excludes JAX-internal frames. The preceding is the original exception that occurred, unmodified.
--------------------
The above exception was the direct cause of the following exception:
TypeError Traceback (most recent call last) /content/min-dalle/min_dalle/models/dalle_bart_decoder_flax.py in __call__(self, decoder_state, keys_state, values_state, attention_mask, state_index) 38 keys_state, 39 self.k_proj(decoder_state).reshape(shape_split), ---> 40 state_index 41 ) 42 values_state = lax.dynamic_update_slice(
TypeError: lax.dynamic_update_slice requires arguments to have the same dtypes, got float16, float32.
pip3 install flax==0.4.2 TypeError: lax.dynamic_update_slice requires arguments to have the same dtypes, got float32, float16.https://www.smithsonianmag.com/smart-news/us-copyright-offic...
> But copyright law only protects “the fruits of intellectual labor” that “are founded in the creative powers of the [human] mind.” COMPENDIUM (THIRD) § 306 (quoting Trade-Mark Cases, 100 U.S. 82, 94 (1879)); see also COMPENDIUM (THIRD) § 313.2 (the Office will not register works “produced by a machine or mere mechanical process” that operates “without any creative input or intervention from a human author” because, under the statute, “a work must be created by a human being”). So Thaler must either provide evidence that the Work is the product of human authorship or convince the Office to depart from a century of copyright jurisprudence.
[0] https://www.copyright.gov/rulings-filings/review-board/docs/...
If you give the computer the instructions, such as “avocado chair”, the avocado chair is yours.
It wouldn’t be yours if it was something like a deep dream- if you ran the program with no input and generated a “random” work.
The report I’ve cited makes a compelling argument against your claim, and several prominent organization’s copyright policies align with it.
Just hitting a "Generate random artwork" button indeed certainly doesn't seem to qualify as "creative output or intervention", but as for how DALL-E and consorts currently work, I'd say that coming up with a suitable prompt text, potentially refining it to get the output closer to what you want, curating the output, maybe using one of the output pictures as input for further processing, etc. arguably all are at least some amount "creative input or intervention from a human author".
Clone before kill.
There is no equivalent for illustrators.
My friends who studied some specific language are all unemployed or doing unqualified jobs. Their peers from a generation before are teachers or work in some embassy.
That said, before some unicorn really start doing some serious polishing, you'll still want some illustrators to piece art together. Taking the output of these models won't deliver a ready made product easily.
I guess lots of artists will move into teaching art.
I see tools like this might increase interest by the public into making their own art with the help of new tools, and some will want to be taught.
You need well cultivated intelligence to obtain a good product: if "anything goes" is the motto, if "cheap" is the "mandate", there lies the issue.
That is, a surface-level view might show these things as equivalent, but the skills required to produce a decent result are not encapsulated in the averages that models contain.
But I'm more bothered by sociatal effects where art is automated. I believe it'll expedite the effects we saw when the internet short circuited the feedback loop for creators, killing any gaps where non revenue optimizing humane creative force could thrive. Not to mention the crazy mimetic positive feedback loops tearing the discourse apart.
Text-to-image algos did the same thing for a while, but you look at the latest full-size DALL-E and it's pretty much flawless.
If I were considering art school, I'd certainly be reconsidering my options. Maybe there are some defects in the output, but nothing photoshop can't fix.
I think where humans win out (for now) is where a high degree of specificity/precision is needed (e.g. graphic design). Or certain legal requirements are present - AI art can't be copyrighted at this time - such as logo design.
But it's not going to tell you in clear words if your prompt was bad to begin with, like a human would, hopefully :).
Further, artists have a host of skills that DALL-E doesn’t, like “take that image, but change the colors a bit to make it more acceptable to the client, and move the cartoon bird a little further down”. Or “make an image that will look as good in a print as it does on a small screen”.