Show HN: I made a new AI colorizer | Hacker News Reader

Show HN: I made a new AI colorizer

palette.fm

472 pointsemilwallner120 comments

Hi HN, I’m Emil, the maker behind Palette. I’ve been tinkering with AI and colorization for about five years. This is my latest colorization model. It’s a text-based AI colorizer, so you can edit the colorizations with natural language. To make it easier to use, I also automatically create captions and generate filters.

Let me know what you think.

You can see some of my results on my reddit page: https://www.reddit.com/user/emilwallner/?sort=top

Open articleView on HN

Thank you so much Emil - I have some old photos B/W of my dad who is no longer with us and have wanted to colorize them somehow for years. The results from just dropping the photos on your page - no tweaking or whatnot - are incredible and have me in tears.

Amazing work.

Comments like this are what make me come back to HN. I hope the memories are kept alive.

Thank you so much. My daughter is away at camp - I can't wait to show her tomorrow too when she gets back.

I can't remember the name but there's a Reddit subreddit where people can post photos to request colorization, so that may be an option if you're curious what some passionate volunteers might do for you.

Great suggestion - thank you.

Awesome job! This makes me a bit envious. I am hobbyist colorizer [1] who did it the old school way (with tools like Affinity Photo and lot of manual work). I tried de-oldify and other tools but I justified my work thinking how horrible they were but pallette.fm is way too good. Not sure if I would find the motivation to restore old photos anymore. Glad that this was just a small hobby for me and I had just started learning the ropes.

But I would be dead scared if I was a professional[2] who did this full-time. Is this what AI taking your job feels like?

[1]: https://shubhamjain.co/experiments/

[2]: https://www.youtube.com/watch?v=vubuBrcAwtY

ty! i'm glad you enjoy the tool. As for whether or not this tool could replace a professional colorist, I think it depends on the specific project. For some projects, Palette could do a great job of automatically colorizing photos, and for others, a professional colorist would still be necessary to get the best results. Especially when the projects require historical accuracy or a high aesthetic standard. it also makes colorization more accessible, which leads to more opportunities to refine results manually or say print the results.

Even if the results are not 100% historically perfect, 99% of the hard job is done. Skin tones, plants, skies, water... they all look incredibly good, and the object segmentation is almost perfect too.

The tool seems to struggle with fabrics, but that part is by far the easiest to fix with a traditional photo editor.

Congrats man. You made my mom happy this evening. Please keep a free tier on your tool.

its fast and looks absolutely stunning but if you've done enough hand work you can still spot flaws (that are easy to correct by hand). The easiest to see is probably the use of black in parts of the face that are not dark shadows. Its slightly under-colored, it had to draw the line some place. It looks like black smudge's left and right of the nose. Then again, few/no one would notice without looking for it.

edit: if a human made these this comment would not exist. It makes A fun angle of AI, it can take criticism and do useful things with it. Many humans fail that challange.

Ten years ago at my first job I used to colorize photos professionally, seeing how this tedious process has been automated away brings me the same joy as when you automate a routine with a shell script.

Could it have been trained on work of people like you?

I gave it ansel adams photos which it did really well on. Then I tried some famous national geographic photos, like the these: https://www.nationalgeographic.com/photography/article/meet-...

still impressed

I decided to go really old, so I tried "Valley of the Shadow of Death" from the 1850s where we get some red and blue cannonballs, but that's fine.

1838 Louis Daguerre photo of paris: https://9ol.es/1838.png

1906 Kite aerial of san francisco after the earthquake: http://9ol.es/1906.png

1895 Montparnasse train wreck: http://9ol.es/1895.png

1920 Wall Street bombing: http://9ol.es/1920.png

1920s Park Row skyscrapers: http://9ol.es/parkrow.png

Nice, maybe I can turn the movie The Lighthouse from black and white into a color motion picture. I hear color "movies" as they call 'em are what's hot these days.

Original: https://i.imgur.com/9CZ7vk1.png

Base Palette: https://i.imgur.com/iTq5H9W.jpeg

It looks pretty great, actually -- https://i.imgur.com/fxZz6f7.jpg (Base), https://i.imgur.com/2JAHnbe.jpg (Vivid Natural)

I tried it as well (edited my comment above), I like the Base Palette option better, it looks more realistic compared to Vivid Natural which looks like a WW2 movie filter with the brown atmosphere.

Very cool.

Your results look much better than the washed out AI colorization that I've seen in the past.

I think you could charge money for this service.

A suggestion for something fun to do / marketing tool: recolorize this video frame-by-frame https://www.youtube.com/watch?v=hZ1OgQL9_Cw

Feature request: colorize B&W comic books. I really want to create a full color book of Calvin & Hobbes comics. (Not for publication)

Thanks! yeah, a few people have used the tool to colorize videos, frame by frame. For example Lord of the flies (1963): https://www.dailymotion.com/video/x8eiho4

Although, I'd recommend colorizing a few key frames and then use https://github.com/zhangmozhe/Deep-Exemplar-based-Video-Colo...

Cool, yeah, my next model will be better for comic books. You can also use the 'Surprise Me' button in the editor and you'll get some decent results.

I ran a test with a photo I took if people are curious. It does a great job for guessing, but of course there is room for improvement.

https://i.postimg.cc/Y9dH4GxW/palette-fm-test.jpg

Cool. Interesting to note that the original is not necessarily the "correct" real world color palette (it's accuracy is subject to camera settings, sensor response, lighting, etc), so the colorized version could in theory be even better than the original in some or all areas of the photo.

As well as there are multiple desaturation methods, GIMP alone offers at least three, and I'm sure that could influence the outcome as well.

https://docs.gimp.org/2.8/en/gimp-tool-desaturate.html

100% I think my original photo is a bit oversaturated. The real issues I see are where it makes blue things brown and red things green.

I love how the emphasis on initial letters are understood and colored red!

Unfortunately, this has the common bias[1] with IA coloring of making old stuff look dull, reinforcing the Hollywood cliché[2]. The past was actually often very colorful.

[1]: https://nitter.lacontrevoie.fr/gwenckatz/status/138165207169...

[2]: see this video by a movie props maker about why Hollywood movies make old things look the way they do in movie https://youtu.be/mF1VFlCnLQ4?t=434

Nice UI, and the approach works quite well (occasional wrong choice of palette compared to ground truth). It’s kind of fun to turn my photos into B/W and run through the model to try and guess the output.

Do you have a privacy policy for the uploaded photos? I’m not keen on uploading anything important without knowing how it’s stored or will be used in the future.

cheers! i'm working on a proper privacy policy. I don't store any images that are uploaded. i use google analytics and mixpanel to store user interactions.

I was actually testing Photoshop's new colorizing filter this week, so this is timely. My first impression of Palette in comparison is that it does a comparably good job of colorizing. Sometimes Photoshop does better, sometimes Palette -- usually the skin tones is the hardest part, and one does a noticeably better job than the other.

The natural language modifications available in Palette ("his shirt is light blue") are super useful. Well done.

I tried it on the photo linked below and it makes the string instruments look like they're made of brass. Not trying to bash it since I thought it was pretty impressive overall, but I'm curious about what leads to this type of failure.

https://commons.wikimedia.org/wiki/File:Jazzing_orchestra_19...

thanks for the feedback! it's made out of two models, one model creates a caption and the second model takes the caption and the black and white image and colorizes it. if you click on the edit button you can see the text that generated that colorization. if the text is incorrect, you can edit the text and recolorize it. this often leads to a better result, however, some cases are still hard, especially damaged photos.

I assume you're using CLIP or BLIP for the text generation of the model, and then img2img or something like that for the colorization. What model are you using for the latter colorization?

> Vivid Natural — A slide of jam, jazzman, hook, band, pipe, and coil. Small contrasting details in natural colors. vivid natural.

This one generates the string instruments with the correct colors.

I've never posted here before but this demands a sincere "thank you".

I just colorised some family photos from the 1930s to the 80s and the results are heartwarming.

Thank you for making this and making it free and, most importantly, easy to use.

Can I paint some colors as a starting point (like img2img) and let it finish the rest?

What about a conversation, like, "the dress isn't blue, it should be orange", doing on top of previous prompt?

not yet, these are great suggestions. it's always a dilemma to add features to mitigate the performance of a weak model, instead of making a better model. most of the problems go away with a better language and colorization model, and many model-specific features are made in vain

I think there are two uses for an AI colorizer. One is to generate a color image that looks great, another is to generate an image that accurately reflects the true color of things.

A better AI model helps a lot with the first goal, but help only so far with the second one. Truth to be told, there is a lot of contextual color information in black and white photos that an AI model can exploit; but nothing beats someone that knows, for sure, the color of the dress of someone in the photo.

I mean, take a look at https://www.reddit.com/r/ColorizedHistory/ - some of those color artists do a lot of research to know the exact shade of green of the military uniform of some country in the 19th century, and things like that, just to have an accurate reference.

So I think that the ability of directing the color output (either by rejecting a color textually, or by painting over the figure with a starting point - even if maybe I'm not painting with the exact tone or texture but a rough color that should help the AI to figure out the details) is essential for a colorization product, even if the model is flawless!

Actually, thinking of /r/colorizedhistory, here's a comparison of a professional colorization and palette.fm https://www.reddit.com/r/ColorizedHistory/comments/y5mfqu/pa... vs https://www.reddit.com/r/ColorizedHistory/comments/y5mfqu/pa...

My concern here isn't that the professional photo has higher quality (in this case it has, but give it some time - months or years - and maybe technology will catch up). It's that sometimes we already know the right color, while the AI must always guess

true, thanks for adding context!

So you can kind of do that, the first step creates a description, and the second step colorizes using the description. So you can modify the text for various degrees of success and specify 'orange dress within the text'

Wow, any way to run this locally?

I love this technology but it would feel kinda terrible to upload a whole bunch of stuff to your site and exploit your generosity.

not for now. i might make a local version in the future, but for now, enjoy!

Yeah agreed. I feel a bit icky running computationally-expensive stuff like this on someone else's dime. Give us a crypto address or something to donate to! Preferably not PayPal, though, if my say mattered.

Nice. Combine with restoration like this https://www.reddit.com/r/MachineLearning/comments/j01y9u/d_b... would bring life to many old photos.

Looks great and works particularly well.

A couple of bug reports: - Clicking on the P logo on the home page takes me to a page with what looks to be an example, but none of the filters ever load. - Navigating directly to https://palette.fm/color/filters returns a 500.

It's funny to let it operate on a pile of random Lego blocks of different colours. Funny how it sometimes uses a completely different colour. In some other cases it guesses it completely right. Sometimes you also see a from one color to another color gradient, while the real thing has one color with a shadow gradient. How large is your trainingsset?

Any chance you would license the model for us to use with pixlr.com?

i'm planning on doing some sort of API, feel free to ping me at emil@palette[dot]fm with the constraints you have

Wow I put a few b+w photos, initially shot in colour so I can compare how it did, and it more or less nailed it—great work. They were shot on a iphone, so I wonder if newer digitally-shot photos have more data for the app to parse, or if it's generally easier compared to film (and whether it uses any EXIF data)

I expect the training to be done mostly on modern digital cameras

Just a quick heads up, when I ran the file and then opened the result in irfanview, I get a warning that the file is a JPG file with an incorrect extension.

You may want to check your encoding settings to make sure everything gels together.

Otherwise, great job! This is pretty nice stuff and way better than I could do on my own!

(Genuine) question, and no snark intended: why do so many people want to change history?

Black and white photos are - for many of us - part of our shared historical record. Is there really a need to improve (=change) them? Can't we appreciate them exactly as they are, without modifications?

Some use picture to imagine how life was back then. Life wasn't black and white.

Same with the facial expressions in old photos. People look pretty serious but that's just because of the technology pf photography back then. People were as silly and joyful as nowadays.

> Some use picture to imagine how life was back then. Life wasn't black and white.

Life (time) doesn't tend to stand still, either, yet we are able to appreciate photographs.

Would we really benefit from old photographs being AI-animated into "videos"?

Perhaps I'm showing my age, but the older I get the more I feel at one with life's imperfections. I'm fine without filters and HDR ... or colour ... or motion.

> Would we really benefit from old photographs being AI-animated into "videos"?

Yeah that would be pretty cool.

>Would we really benefit from old photographs being AI-animated into "videos"?

Some people seem to think that

https://news.ycombinator.com/item?id=26279372

I think that the filters, HDR, or the guesswork restorations of old media are similarly life's imperfections - just that the life is more current, compared to the original date of taking these photos.

> People look pretty serious but that's just because of the technology pf photography back then

This is a known misconception. People on old pictures are not smiling because they liked it this way, not because of long exposure times. Same for paintings. Except for Mona Lisa, you were supposed to be dead serious when being immortalized on picture for generations to come.

Sounds similar to police mugshots, you don't want to be smiling or too happy as future generations may mock you...

Most people see the life in color and are not used to black and white images. Color give way more depth and understanding.

Nothing is being changed. After using the tool you still have the original, plus a new version which can be used alongside

The impulse to colorize monochrome photos goes back almost as long as photography itself.

https://en.wikipedia.org/wiki/Hand-colouring_of_photographs#...

It depends on the photo, obviously, but colorization can also give a new dimension of contrast to a photo, even if the colors aren't strictly accurate, by separating out the foreground from the background better.

> The impulse to colorize monochrome photos goes back almost as long as photography itself

Impulse? A strange word to choose. Impulsive behaviour isn't what I'd aspire to, or want others to aspire to. Be thoughtful.

Who are you scolding and why?

I was using it in the sense of definition 1b, a propensity or natural tendency usually other than rational: https://www.merriam-webster.com/dictionary/impulse

People have liked colorizing photographs for a long time, it's a natural tendency even though it may or may not be rational.

To me at least, it's just entertainment. Do you find it offensive to the past?

>Is there really a need to improve (=change) them?

Yes, there is. They offer a specifically degraded perspective into the past - namely, the loss of color. I do appreciate them as they are, and I also think that the effort to use them as inputs for current technology is also very interesting. I think that seeing these photos, the past in color, helps to humanize the subjects of the photo, and see them, and their environment in a more accessible, realistic way.

I'd like to present this collection of real color photographs that date far back - I think they are really interesting to look at, because we're so used to the black and white version of the clothes, technology, buildings from back then.

https://www.buzzfeednews.com/article/gabrielsanchez/oldest-c...

This works amazingly well, just putting in some of my favorite black and white scenes from movies... wow! Now I need some kind of tool that can do this on every frame of a black and white movie. I cannot wait to be able to watch some old classics fully colorized.

Thanks, this is awesome. Colorized a few last images of my WWII grandfather who died in this war. One observation: on download the MIME type seems to by PNG format, but the binary file arriving is actually a (lossy compressed) JPEG. But anyhow! Great work!!

I had some black and white photos of my passed-away tuxedo cat, fully expecting this to produce a terrible result, but instead I was blown away by accurate and subtle colors added. Pinkish nose, ears, green eyes all came out of the photo.

Wow! I am impressed. I uploaded a WWII photo of airmen posing in front of their plane and got an amazing result. I’m not certain if WWII era life vests were bright yellow, that seems out of place, but it’s a good guess if not.

Found some that are yellow, not sure how bright we're talking - https://www.ima-usa.com/products/original-u-s-wwii-usaaf-mae...

Jaw dropping, that's the approximate shade it used in the picture.

Here is the B&W photo I tested with (I think "base" or "ambient historic" palettes are best):

https://1.bp.blogspot.com/-ppnklwWgzIE/TYub-kkx8cI/AAAAAAAAA...

Here is an similar, actual color, photo:

http://4.bp.blogspot.com/-gzlivCEVuMk/VPzsJUVlYMI/AAAAAAADm_...

It also did really well with the different skin tones of the Tuskegee Airmen:

https://i.pinimg.com/originals/21/7e/16/217e161c9e6cfe74861b...

It did not do so well with this grainy photo of soldiers w/ a captured German flag:

https://i.redd.it/hbus98dh8sfz.jpg

If you use the "vintage charm" filter it does a good job with the flag raising at Iwo Jima:

https://blog.uspatriottactical.com/wp-content/uploads/2020/0...

Thank you for providing pre-set examples on your site. That's one of the major barriers on these types of tools to "test" especially when you're not sure where/how/if the images are public.

I immediately was limiting this to think it would just colorize B&W images, but you can use this on images with colour to use natural language to modify the colours of an existing image too!

Open Source? Would love to see how this works and also the website.

Today I was thinking of some sort of NN trained for colorizing photos, and then I get here and there's an AI right there...

This tool works really well. The text prompt helps, too. The suggested colors leak to surrounding objects (I forced a red skirt on a traditional costume I knew was probably red), but it looks really good.

Did you decolorize pictures originally in color in order to train the model?

The ease of use,speed and accuracy are impressive. I tried doing a colorization by hand a few years ago using photoshop. It took a long time and it was not as good.

Over the years, I've read that a products that's 10x better will be a game changer. I think you have, at least, a 10x better product. Congratulations!

I'll be using your service for my B/W prints. Great job!

This is great. I'd also love to have the option to manually draw guiding colors over parts of the image, for even more control.

All I can say is: very very nice (Testing on some 3d renders and some bw photos). p.s. Is output image width limited to 1920px?

ty!! yes, 1920x1920. i'm working on an unlimited resolution option, but it will take a few weeks.

I've tried a few of my own black and white photos and the description of each starts with "a stock photo of a..." Is this a comment on my style of photography or is this always added for the base palette prompt? (The colorized images have been very impressive.)

Excellent work - enjoyed the rick roll. Would you mind sharing how you've done the GPU deployment?

lol, thanks! onnx, docker, and fastAPI on CPUs with AWS fargate. although i'm switching to GPUs in a month or two, so if you have any suggestions let me know.

I've also been searching for GPU solutions for potential small projects like this, and so far banana.dev and runpod.io seem promising.

Incredible service. Just annoyed once I show it to my folks I'll be dragged into multi hour tutorial session on how to scan / upload old albums. Also curious which "filter" they feel will most reflect the subjective reality of the past.

Heh, a lot of the results are more visually appealing than the real landscape I shot in B&W.

I love the way it finds features and labels the picture. It thinks I am a meteorologist!

This is magic… How the heck does it know that the walls of my home gym is light green?!?

Impressive!

Quite an incredible compilation of photos on your reddit page, thanks for sharing.

thanks, that means a lot!

I thought from the title it was some JavaScript to auto-colorize essentially any programming language and was excited, because that stuff is annoying to implement for any new programming language out there.

Still cool though.

This is absolutely mind-blowing to me, is this using an API of any sort? I'd love to be able to take something like my City's archive of Black & White images and colorize them.

Wow, it’s awesome! What technologies does it use?

I’ve used to restore the only photo my wife has of her father who died when she was 3yo. She almost cried seeing the colori ed photo, thank you

It would be nice if you could put hex colors in the prompts. E.g. instead of saying "man wearing blue shirt", you could say "man wearing #0010e4 shirt".

Amazing. Colorized a rare picture of my grandfather. Thanks!!!

Colorizing b/w photos is one thing; these are limited representation of colorful reality.

Now let's colorize b/w works of Albrecht Dürer, or Francisco Goya.

I tried one of my recent B&W film images. It basically recreated the original scene (a graffitied building). Nice!

Tried a few old family photos and it's done a good job - mostly better than the other ones I've tried. Excellent work.

This is incredibly well done! Congrats. It’s such a solid layer on top of recent developments and offers instant value.

Wish you much success!!

I tried some of my old university photos. Totally amazing. Thank you so much for a wonderful work

I haven't tried any other colorizer tools but this one works very well for me. Thank you.

that's incredible. congratulations on the great work and thanks for sharing with us.

How are the prompts/captions generated? Also I love that you're using Svelte!

This is a top tier side project.

Can you summarize the approach you took on creating the model? Really curious!

This is a really impressive colorizer. Don't feel guilty monetizing.

I laughed out loud reading your ABOUT page. Thank you!

First time getting rick rolled in plain text :)

This is wonderful. Works really well. Thank you!

Is there a way I could contact you directly?

How does it work? Do you write scientific papers?

I haven't made a write-up of this yet, I still need to figure out how to self-fund it. I normally do more layman scientific writing like this: https://artsandculture.google.com/story/the-klimt-color-enig...

Amazing work! Thanks for sharing!

This is quite amazing, and useful. Great work.

How does this differ from deoldify?

tested on a few pics. excellent colorization. pls provide patreon etc link so the site stays up! thxxx

very cool. just coloured some old family tree pics and they look great. thanks.

Amazing work congratulations!

Hi Emil, thank you for this!

This blew my mind. Amazing.

Wow.

Reply on news.ycombinator.com