Improved seam carving with forward energy

You're absolutely right that RGB differences are not very meaningful to humans. I chose to stick with the easiest approach for educational purposes, but I'd love to hear if anyone tries out a different color difference function.

Note that the authors tried other types of energy functions in their original papers, ones that didn't simply use color differences. No one energy function seemed to be perfect across a variety of images.

Completely agree with this. In my experience, LAB space really makes an observable difference in perception-based applications, as compared to the mostly arbitrary RGB cube space.

RGB values in images are de-facto sRGB, which is reasonably close to human perception (sRGB gamma curve is close to Munsell lightness curve).

In this case applying gamma correction to compute the color difference in linear light would make things worse, as it'd model physical properties of the light, rather than human perception.

I must have missed the real-time adaptation of seam carving, so thanks for bringing it to my attention! I'll certainly take a look at it.

You can do something like that. For example, enblend does multi-scale "blending", where it builds an image pyramid and runs seam carving on each level before re-combining.

Never did it in school, but when I came across it as an assignment from a Brown CS class I decided to give it a go on my own, since I'd done Racket stuff at school and new my way around it decently enough. I agree wholeheartedly -- it is just insanely cool.

Literally everything about it. The concept, visualizing intermediate steps by producing a picture that shows the energy of each pixel, then the CS with finding the path, then seeing the image it produces. Just an incredibly interesting (and very modern issue given different screen sizes), a clever algorithm, challenging to implement, visually interesting results... It literally has it all.

In the first paper, the authors discuss what happens if you take the lowest energy pixel from each seam. The problem is that visual coherence is not maintained, because there is no connection between the removed pixel.

What you're talking about--expanding the number of pixels the seam can move from row to row--is certainly possible. This will require more computational resources, as each seam can continue from five different seams (or more if you choose to widen your slope parameter). And, I'm not sure you'd get enough visual coherence from these types of seams.

That said, the best way for to know for certain to is to try it out!

The algorithm is re-run, but is usually implemented using a dynamic programming approach to avoid completely recalculating every seam from scratch.

Good question. I talked about it in my previous article, but the gist is I compute the energy (or costs in this case), then use dynamic programming to find the new lowest energy seam each time. This allows a subsequent seam to pass through an earlier seam, as seams are found based on the latest images, where pixels have been pushed together from previous removals.

However, you'll notice that the costs are only changed in a small area around the removed seam, so if I optimized the process, I could avoid recomputing costs for the majority of pixels.