Over-engineering an emoji webcam filter with a neural network | Hacker News Reader

Over-engineering an emoji webcam filter with a neural network

jott.live

79 pointsauraham18 comments

I suspect calculating the average RGB of each emoji and comparing the distances between the average RGB of subsections of the image would have resulted in better emoji choices and better output (simpler, too).

I mean, I get that maybe it’s not the point of this article, but I think the end result would have been a better ‘mosaic’.

Rgb isn't a great color space to use with distance measures. CIELAB is better, though really perceptually uniform color difference measures are still nontrivial in that space as well.

Take a look at the ‘12’ on the bottle of scotch. Higher resolutions than the dimensions of the emoji pixels would otherwise warrant are achieved by selecting emoji that work akin to subpixel antialiasing.

I'd reckon there's probably room for about 2x2 subpixels' worth of "antialiasing", making the search space 4x3=12 dimensional (probably in Lab colorspace). Is that doable? If yes I'd expect the approach to be close to optimal possible.

I think there is an algorithm for determining the dominant color in an image (in this case, in a emoji). I do not its name, though.

https://stackoverflow.com/questions/3241929/python-find-domi...

https://dev.to/producthackers/creating-a-color-palette-with-...

You mention a lookup table would have been tedious, but surely an easier method.

This might be a case of "to a hammer everything looks like a nail".

The solution in my familiarity would be to align the emoji in a grid, take one screenshot and write a script to iterate over the pixel data to figure out the most dominant color and how dominant. So the build of the lookup set would be automatic.

Less interesting for sure but easier (at least for a JS dayjob skillset).

Similar to a below comment, I know this isn't the point. Thanks for an interesting write up

I think a (N, D) array can be used as a lookup table. Here, N is the number of emojies and D is the dimensionality of the color space (say 3 for RGB). Each emoji is converted to a D-dimensional vector by computing the dominant color. Then, we can take a D-dimensional vector from an input image and compute the closest emoji by using cosine similarity. To minimize the loss function, we can increase the number of dominant colors, so the table would be (N, DxM). In this case, each emoji is represented by M D-dimensional vectors, each one encoding a dominant color.

Do you have any good(preferably light on dry math) resources on these kinds of multidimensional search problems?

Forgive me if I'm misunderstanding how it works, is the model is predicting the original emoji from an augmented emoji, and doing that for every 36x36 patch in the webcam image?

author here! I grabbed them from here IIRC https://emojipedia.org/apple/ (there are a bunch of sites that have these)

edit: yep - predicting the original based on the augmented version (as a hack to generalize for unseen inputs)

I've updated my comment to clarify the question

Nearest neighbour algorithm though, no?

Yep, isn't that used to find dominant colours in images fast anyway?

Hey, Bun in the wild!

does Bun just wrap node.js?

I believe Bun is meant to replace Node. Could be wrong though, I am not a Javascript understander by any stretch of the imagination.

I think author can try to encode location of the emoji and incorporate better scoring function (GAN)? But that is obviously really "over-engineered" :)

ai already lost my guys

Reply on news.ycombinator.com