Hacker News Reader

Comment by doctoboggan | Hacker News Reader

For those with experience with ML in Julia I’d love to get some advice. I have a little business that sells 3D printed jewelry (https://lulimjewelry.com). My biggest seller is customers engraving their or their loved ones fingerprint on the ring. Most of those prints come in needing manual cleanup, which I can usually do in a few minutes.

I’d love to train a ML algorithm to do this, and I’ve been building up the before and after pictures over time using my manually cleaned up customer fingerprint images. Can anyone give me suggestions or pointers on the sort of algorithm that may be best suited for this task? Just something to get me started down the correct path would be very helpful.

View on HN

Depends on what you are cleaning. I have a hunch you wouldn't even need anything ML related. Just the right computer vision pipeline using OpenCV should get you really close. If you really want to use ML/DL because its cool and fun, I'm pretty sure Autoencoders will do what you need [1]. If you're lucky, the denoising autoencoder should do the clean up automatically. Alternative is to use a VAE which goes really far compared to GAN since its much more sampling efficient and easier to train. I think(?) it's what was used in the original deepfake.

[1] https://www.mygreatlearning.com/blog/autoencoder/

You're right on the VAE. Depending on output quality, my experience has been fewer samples are needed for VAE training.

I agree. For cleaning up a fingerprint I would think you could get good enough using Opencv. People have been cleanly vectoring raster images long before deep learning.

If you can automate it using a combination of Gimp filters + Inkscape, you probably can do it pretty easily convert that process into code.

I can and do simply rasterize images for some of my clean and proper fingerprints. The problem I am having is with smudges, incomplete prints, or other artifacts. In those cases I trace over and fix the print sometimes taking a decent amount of artistic license depending on the underlying quality of the image.

maybe a quick mvp would be to detect smudged images, and separate those out for manual handling, and the non smudged automatically rasterized.

I'd also take a look at Images.jl: https://juliaimages.org/latest/examples/ Noise removal example: https://juliaimages.org/ImageFiltering.jl/stable/democards/d...

I can't really share any images with you as its my customers PII, but the cleanup stage involves a decent amount of artistic interpretation which is why I didn't think a normal image processing pipeline would work. They often come in smudged, missing lines, incomplete, or with some other error. Its not just a matter of playing with curves, levels, brightness, contrast, etc. I have to trace over them defining what I think the underlying print probably looked like.

Naively that doesn't seem like a major problem. I'd see it as largely setting up a bunch of constraints and methods to generate candidate solutions - after the initial image processing, obviously.

How far could you go with some automatic cleaning, something like `maptrace` [0] and some topology rules when you have your vector file to finish with some manual artistic polish? Because, i don't believe that you will achieve easily the "do-it-like-me" result expected from either GAN or VAE or autoencoders. In case of you want to look, the classic examples about style transfer (Photo to Van Gogh style, etc.) seems to be the way to go. Specific to Julia, you can look at Flux.jl or KNet.jl.

[0] https://github.com/mzucker/maptrace

I think it's a common misconception that OpenCV is Photoshop in coding form. There's a ton of powerful that you can use for what you described but hard to say without seeing the pics. A good example is noise removal in the PSD domain (power spectrum density, not PSD photoshop files).

[1] https://docs.opencv.org/master/d2/d0b/tutorial_periodic_nois...

ML practitioner here.

Step 1: write out the steps you do to manually clean up.

Step 2: Depending on: (1) how many samples you have, (2) what is defined as "before" and "after", (3) what performance level is acceptable, there are many algorithms to choose from. Transform learning may be the target, reinforcement learning or CNN as well. It really depends on Step (1) and (2.1).

For a given poor quality print I usually take it on my iPad and retrace either the whole print or just a portion. I usually just use the pencil tool in white and run it between the ridges of the print to give good separation and definition to the final engraving. Depending on the quality of the print (some of them are hardly more defined that a low res ink smudge) I may have to use a good amount of artistic liberties on where I think the pattern was going. Using my knowledge of what a print may look like I can usually do a decent job. Thats why I thought this might be a good task for a ML algorithm since I can show it what a bad print looks like and what a decent cleaned up version of that print should look like.

So you're denoising by identifying where the patterns should be, but there is some smudge?

If so, you may be able to skip ML altogether and use a deterministic software program to classify light versus dark areas (what you're doing with your pencil). ML might help with mapping smudges to the right shape/sizing.

That said, if you have a ton of examples, then yeah an image recognition task might help.

Also: https://towardsdatascience.com/fingerprint-denoising-and-inp...

Wow, this is basically exactly what I needed, thanks! Seems like the key word I was missing in my searches was "inpainting"

Glad to hear it.

I got their model running with the included training data and it’s quite impressive. Some of my more extreme poor examples don’t produce reasonable results but many do.

That's awesome! Happy scaling!

I would go for decrappify: https://www.fast.ai/2019/05/03/decrappify/

GAN's are hard to train for a new ML practitioner, VAE can have blurry results if you are not using skip-connections like in U-NET.

It sounds like you have training examples (images pre and post processing), but would benefit from an approach using transfer learning since you don't have that many.

Good luck!

How do skip links help with blurriness? I thought they are mainly to prevent disappearing gradients?

Sounds like you want to convert raster images to vector. https://vectormagic.com/ is a commercial version.

Thats not actually my problem. For a good quality print I can do this with no issue. (I use imagetracer.js in my web app) but the problem I have is many prints that come through smudged or otherwise damaged. In those cases I have to retrace them, sometimes taking a decent amount of artistic license using what I know about how other prints looks.

How many samples do you have of smudged prints that have been corrected? Can you also artificially smudge prints to create a training set? Then an interesting solution might be possible!

I currently only have about 100 fixed up image sets. One of the other commenters pointed me to a blog that mentioned this software for generating realistic finger print images: https://dsl.cds.iisc.ac.in/projects/Anguli/ which could then be artificially smudged. There seems to be a dataset out there somewhat that has already done this (https://competitions.codalab.org/competitions/18426) but I can't find out where to download it.

I would advise against completely synthetic prints, you want to model your training data as close to reality as possible! Can't you just blur other finger prints you have?

Reply on news.ycombinator.com