Fred's ImageMagick Scripts | Hacker News Reader

Fred's ImageMagick Scripts

fmwconcepts.com

1,149 pointsprimogen100 comments

Seeing tools like this pop up periodically and get so many upvotes it seems a lot of people aren't aware of great tools. It would be interesting to see a gallery of Unix classic/modern tools presented in a structured course. Is anyone aware of one?

Something like images (imagemagick), video (ffmpeg/vlc), audio (sox), phone (asterisk/etc.), SMS (pdus/various), linguistics (various), classification (NNs), version control, snapshots, clustering (PXE/corosync/pacemaker), security (kernel toolkits), transaction systems and databases (sqlite/RDBMS/noSQL/time series), networking (filtering/firewall rules, intermittent connectivity, local multidrop protocols, tcpdump/wireshark), kernel security toolkits, CI/CD, etc.

Scope would be essentially everything except tools lying within the popular web stacks.

The goal would be that a student with interest could work their way through the syllabus and emerge a capable multi-domain unix hacker, rather than needing to encounter these problem domains over decades of career, rediscovering well worn approaches.

It would need to be automatically collated by an AI with access to the Internet Archive that is probably quite a bit more sophisticated than what we currently have. DMOZ is the closest curated thing I can think of.

DMOZ seems to have some good resources, going through it quickly I found http://www.tldp.org/LDP/abs/html/index.html which has a very good bash intro!

Not sure why I was downvoted. It's not as if the person downvoting me provided a better option than I did!

YES THIS PLEASE.

I learned programming through the web, so I lack a lot of knowledge in basic systems / bash / *nix / I'm not even sure how to call that.

I've googled so many time "learn bash" etc but it's never really what I'm looking for.

I regularly discover tools that are so basic yet so powerful (all things you mention) yet I feel there is a lack of (great) resources online to "learn the basic tools of your CL".

You mean you don't just go read the man page for every bash command and intuitively understand it?

Sorry for the sarcasm, I just feel your pain. Learning even basic things in bash has been slow for me too. I love it. I prefer it to GUIs on most things I have figured out.

But there's so much to learn.

from a sibling comment: http://www.tldp.org/LDP/abs/html/ seems to be great :)

It isn't as good as: https://mywiki.wooledge.org/BashGuide (which is the one that gets recommended on #bash @ freenode). Also: https://shellcheck.net

I spoke about bash and this exact thing here:

https://www.youtube.com/watch?v=pb3k0sGKrjQ

Plug for book that resulted:

https://leanpub.com/learnbashthehardway

Or learning git from its man pages.

https://www.mercurial-scm.org/pipermail/mercurial/2015-March...

UNIX Power Tools, written in 1992, once filled much of this niche. It badly needs updating and augmenting, or a novel replacement.

Well, the way this person uses ImageMagick can hardly be described as "basic". In fact, he's a wizard. If you look through the bullet points at the top, he links to an impressive CV and PDFs with mathematical descriptions of how some of his scripts work.

You can scroll through /bin, /usr/bin/ and /usr/local/bin/ and find all the included basic tools and then read "man 'tool'" or google for it

This is not practically useful advice.

It's how I learned, back in dial-up days.

Coreutils is a good place to start.

http://www.gnu.org/software/coreutils/coreutils.html

I’m not saying you can’t learn this way. I’m saying it’s not practical for most people. Many people who would fail this way would succeed other ways. Reading man bash as primary learning material will lead to failure for 999/1000 humans.

That just means the man page you're using sucks.

I’ve wanted to read man bash, and believe me, I’d think I was the kind of person who would (I learned Go just reading the spec. It’s a great spec), but man bash and many other man pages have never worked for me. Maybe if I tried again now.

bash, incidentally, has a man page that sucks. I've found that most larger GNU projects have pretty horrible man pages. Probably due to feature creep in said projects - so the manuals have to be large to cover all the features.

I spent 10 years avoiding awk, because every time I typed 'man awk' I was overwhelmed by the GNU monstrosity. Then I found the man page for the plan9 implementation of awk (http://man.cat-v.org/plan_9/1/awk) and learned the language in fifteen minutes.

> I've found that most larger GNU projects have pretty horrible man pages. Probably due to feature creep in said projects - so the manuals have to be large to cover all the features.

Rather, info manuals are GNU's official (self-invented I think) documentation system. They make manpages only because of their popularity, but they direct you to the info pages for more info. When other projects have too much documentation for a manpage, they split it among several pages, like perl, openssl, zsh, git, borg, btrfs, etc. Personally, I like both, and wouldn't rank one above the other.

Try 'info bash', it's a lot easier to browse. 'info' by itself gives you a menu of installed info documentation. 'C-h i' in Emacs.

It's far better than Googling or guessing for them. Looking in those places provides exactly what is on your system and the man pages exactly describe what they are for.

If there is a better source, I don't know it.

That's how I learned a lot of things; by literally scanning through everything it had to offer and googling what I didn't understand. I did that for linux utils, language references, most programs I use and most codebases I had to work on.

One notable example is the time I had to learn C# 4 years ago. It took an entire morning to scan through the entire language reference and then later I found out I already knew a lot more than "advanced" devs who would just look up stuff as they needed it, without knowing what actually is available in the first place.

I feel its the only sensible advice if you want to learn a lot on your own with limited time.

My experience has been that it’s most important to get started. Then you can scan and read man pages etc.

Yes, it is practically useful advice, especially when you’re on a flight with no WiFi.

That’s a good use case. Believe me I’m into man pages. Every 6 months I check the Apple store to see if anyone has created a good man pages app. But as primary learning material for most people: not so sure.

I never thought of that, and said to myself, "Hmmm... that is a good idea."

That is exactly how I learned how things worked.

Explains concepts: https://www.youtube.com/playlist?list=PLFAC320731F539902

If you can spare the time read these two, the first one is free.

http://linuxcommand.org/tlcl.php/

https://www.amazon.com/UNIX-Linux-System-Administration-Hand...

The best way to learn in my opinion is to find someone who knows more than you and have them be your mentor. You don't even have to ask them directly, though many people I think are glad to teach.

Identify someone at your work or school who you feel always has a nice perspective and who you trust; then seek out their opinion and advice when you have questions. You can learn a lot in a fairly short amount of time by applying this practice.

You can also find what I've heard called "silent teachers" - there's no formal or informal relationship between the two of you, but you're always watching what they say, observing the things they care about, etc.

If you're in a situation where you're not employed full-time in a technical position it becomes a little harder but you can still do it. Much of it depends on your local context.

Such galleries in my opinion already exist and simply require an afternoon hitting github for "* awesome (list)" with every topic you've listed. Then devoting enough self-teaching time to learn the material. These lists tend to be curated by domain experts and are already in a structured format suitable to the domain.

Here's 1 such example from the 1st result from a Google search for "github awesome security list":

https://github.com/sbilly/awesome-security

Hmm just tried it for audio and nothing I found in three audio-related lists off the curated awesome list of awesome lists even mentioned sox. I do think you suggest a good search strategy in very new areas, but I don't think it's quite the same thing as presented in the grandparent.

Personally I discovered a lot of great tools in my distribution's package management database.

You're right, not enough people know about sox. I can spend hours making random sounds with the synth just for fun. I've been contemplating making something like dwitter.net for sox as a side project. People would submit sox commands for various interesting sounds.

have you tried vips as an alternative to imagemagick? it's pretty amazing.

Never tried it, but it looks interesting where performance and memory use are issues. https://github.com/jcupitt/libvips Also it seems to have a GUI https://github.com/jcupitt/nip2 Thanks for sharing.

If you work in something related to physics (energy, maths, chemistry), I can not recommend `units` enough.

As a small example:

  You have: 10 kcal / g * 24 kg * .13USD/kWh
  You want: USD
  * 36.261333

It's so useful when converting between different units

I want a tool that does that kind of conversion for me. I've been looking(not actively or with any sort of effort) for such a too since my friend showed me a mac only text based converter that did something like the above. I've been toying with writing one.

maybe this list has an option for you? https://alternativeto.net/software/soulver/?license=free&pla...

NaSC looks nice.

> It would be interesting to see a gallery of Unix classic/modern tools presented in a structured course. Is anyone aware of one?

Back when I used Gentoo, I discovered a lot of interesting software by simply browsing their portage repository and installing packages that sounded interesting. It's organized into sections like you describe. There were (probably still are) web sites to browse the package database too. FreeBSD's ports are organized in a similar manner.

A good book for learning the Unix basics is "The unix programming environment" by Kernighan & Pike. This is an old book but the concepts it teaches are invaluable even today. We used it in our university around 20 years ago but I still recommend it to unix neophytes!

The whiteboard script is particularly interesting: http://www.fmwconcepts.com/imagemagick/whiteboard/index.php

Nice! It could be used before feeding a OCR

Wonder how close this is to the techniques apps like Office Lens use. I am always amazed to the extreme angles I can take a photo and the app still makes it end up as perfect pdf.

Check out how Dropbox does it: https://blogs.dropbox.com/tech/2016/08/fast-and-accurate-doc...

Thanks. Exactly what I was searching for.

https://news.ycombinator.com/item?id=12254517

> https://officelens.uservoice.com/forums/289714-office-lens-u...

It seems Office Lens depends on the online servers and doesn't do anything offline.

I still prefer offline solutions, especially open-source ones.

Sure. This wasn’t an advocacy for any tool. I suppose most tools use similar tricks and was wondering if it was anything like the one presented here.

I can't seem to find the link now, but there was a post here a couple of weeks back about someone doing a very similar thing (in Python, IIRC) to clean up pictures of handwritten notes. It seemed conceptually simple, and worked quite nicely.

I know the one you mean, this classic one was linked from it: https://mzucker.github.io/2016/09/20/noteshrink.html

That's the one, indeed. Thanks!

https://news.ycombinator.com/item?id=16567275

Oh man this page brings back some golden memories! I remember working with ImageMagick for the first time in 2010 or 2011 when I wrote a bash script to programmatically scale and center thousands of images for creating image gallery thumbnails on an internal social platform for winners of a design award. I somehow landed on this page and used the massive amounts of knowledge to learn how the ImageMagick scripting interface worked. Then it was a fairly simple task to build the script, but - like all bash scripts for me - I've forgotten every single parameter now :)

Somehow that feels much longer ago than just 7 years, maybe its just the website or the feeling of nostalgia looking at it.

fisheye2rect is just mind-blowing

http://www.fmwconcepts.com/imagemagick/fisheye2rect/index.ph...

I guess I never really understood why fisheye lenses were so valuable, until now. =)

Yes it is. I've been using that script to create 360 photos which can be fed into Google's web vr view. [0]

[0] https://faa-aviation-data-portal.github.io/routeDemo/camera/...

This is really cool, can you explain more how you did this? :)

I don't have the script on hand at the moment. But essentially I take two 180 degree fisheye photos one pointing straight up and one pointing straight down. (for this, I used a homebrew cardboard box and two mobotix cameras)

I then use OP's script to "unroll" each fisheye photo. I then stick them together top to bottom with looks like this:

https://faa-aviation-data-portal.github.io/routeDemo/camera/...

Which I feed into this:

https://github.com/googlevr/vrview

voila

took a couple hours start to finish.

Huh, I remember as a teen bring blown away by IPIX ('90s). It dewarped a fish eye image so that you could look around a scene from a fixed vantage point.

For those interested we implemented from scratch a subset of these commands as a REST API service available to call at request from any programming language.

https://pixlab.io/cmdls

This is a bit far from what most would consider REST. Just call it a HTTP api?

Looks cool though! I would love a product that behaves like a proxy/CDN for images and does some on-demand resizing/filtering.

https://www.imgix.com/

Cloudinary is this exactly. It’s great! https://cloudinary.com/

https://sirv.com

https://github.com/beetlebugorg/mod_dims/wiki

Backed by Imagemagick. We usually run it behind a CDN like Akamai or Cloudfront but you can also use Apache httpd’s built-in file caching.

There is no true REST API. Just like there is no true Scottsman.

I don't disagree with you, but this one in particular seems very far from the definition even if you ignore all the hypermedia stuff.

It would be interesting to augment an image recognition dataset with additional inputs generated by minor transformations under these scripts.

This is fairly standard practice (although I don’t know if ImageMagick is specifically used).

https://github.com/aleju/imgaug - Image augmentation for machine learning experiments

Interesting to see this re-surface on Hacker News. This set of scripts has to be among the most useful things I've found on the internet, and I've used some of these in many of my projects!

These are great and I need to start using these. For instance the dominantcolor script extracts color schemes from pictures. Something I've tried to do by eye in the past with limited success.

http://www.fmwconcepts.com/imagemagick/dominantcolor/index.p...

There is some very powerful stuff in here. I wonder if the author has implemented these in any DCC plugins?

That awk Mandelbrot script is pretty hardcore.

If ImageMagic isn't your thing and like a GUI but don't like Photoshop or the GIMP I really recommend PixInsight (http://www.pixinsight.com/). It's normally used for astronomical image processing but it's well build with some advanced stuff in there. Unfortunately not free, but people have to pay the bills!

No no no, we can't have this, that's too old fashioned!

Proper way to do this is to create an image processing SaaS around these scripts, announce to store the filter presets on blockchain and create a marketplace platform for filter creators, have a $50M ICO, spend the cash on lambos, and fail awesomely.

EDIT: Ok, I just didn't scroll far enough :) https://news.ycombinator.com/item?id=16668686

You forgot to include deep learning. I’m not buying your product!

It constantly amazes me how huge the variation is between some software services and the reward for their effort.

On the one hand you have giant startups earning millions from a tiny SaaS tool.

And on the other hand you have a single developer sharing a huge library of useful tools for free.

It's a crazy world now, it seems opportunity is everywhere.

Old and gold

It's listed alphabetically and difficult to know what each script does. I had to click each one to find out what it does, but these are my favorites:

* tshirt- Transforms an image to place it in a region of a tshirt image

* tileimage - Tiles an image to a given size with various tile arrangements.

* shadows - Applies drop shadows to an image.

* grid - Superimposes a set of horizontal and/or vertical grid lines on an image.

* lupe - Applies a magnifying glass effect in a local area of an image.

* multicrop - Crops and unrotates multiple images from a scanned image.

* peelingpaint - Applies a peeling paint effect to an image.

* picframe - Adds a picture frame around an image.

* picturefold - Applies a map-like folded appearance to an image.

* PINBARREL - Corrects or applies pincushion and/or barrel lens distortion to an image.

* PUZZLE - Applies a puzzle like effect to an image.

* RANDOMCLIPART - Randomly distributes clip art over the image.

* SKETCH - Applies a sketch effect to an image. SMARTCROP - Automatically crops an image to a given size around the hightest detail region.

* TILER - Converts an image into a tilable texture.

* UNPERSPECTIVE - Automatically removes pespective distortion from an image.

* UNROTATE - Automatically unrotates a rotated image and trims the surrounding border.

* VINTAGE1 - Applies a a classic vintage effect to an image

* DIAGCOLLAGE - Collages three images in a diagonal orientation.

* DEFISHEYE - Corrects for fisheye distortion in an image.

* EMBROIDERY - Applies an embroidery effect to each color in an image.

* DOMINANTCOLOR - Computes the dominant color in an image.

* OVERLAPCROP - Creates a sequence of cropped subsections permitting optional overlap of the subsections.

* TRIMMER - Trims the background from any number of specified sides of an image.

* outfit - use texture image onto image of clothing

* splitcrop - split one image into many pieces

* whiteboard - Processes a picture of a whiteboard to clean up the background and correct the perspective

* postagestamp - convert image into stamp

* glow - add glow effect

There is a summary page of what all the scripts do [0].

0. http://www.fmwconcepts.com/imagemagick/script_list.txt

I love ImageMagick but I can't believe there's only one feel facto cli image processing tool. Do people know of others?

There is gmic, http://gmic.eu

It is quite powerful though somewhat eccentric.

I once developed some scripts to use it to "develop" jpg from raw images, see https://github.com/claes/gmic-extensions

I second that. G'MIC has an impressive bunch of built-in commands to manipulate images, see e.g. the examples shown in the reference documentation (click the first magnifying glass, then use right arrow on the keyboard):

http://gmic.eu/reference.shtml#subsection15

And as the commands are built-in, no needs for external scripts, those effects are available from any interface using the `libgmic` API (including C/C++ programs).

graphicsmagick is a fork that’s been going for a while

I prefer the PHP version of ImageMagick as PHP is a better scripting tool than 'cli'. The manual is a bit hard to follow as things are essentially undocumented. However, if you want to carefully prepare a batch of images, going to PHP is taking it to the next level.

Particularly interesting are features such as being able to save images in different colour spaces, e.g. 4:2:0, going lo-res on the chroma for a lower ultimate file size in jpg.

If I had a lifetime spare I would complete the PHP Imagick documentation and write a version of Fred's scripts in PHP, so they can be chained together in a nifty object oriented way...

I agree. I think ImageMagick is available as a library too for other languages

There is the netpbm toolset:

http://netpbm.sourceforge.net/

I use VIPS

https://jcupitt.github.io/libvips/API/current/using-cli.html

Much better performance and better image format support for my use.

It also has a gui

https://github.com/jcupitt/nip2

The GitHub wiki for VIPS has a page [1] very relevant to GP's question. It lists two dozen alternatives to VIPS (both command line tools and libraries) and compares how they performed on a load-crop-scale-apply convolution-save task. The source code for each implementation of the task is right there on the page.

[1] https://github.com/jcupitt/libvips/wiki/Speed-and-memory-use

Thanks!

http://github.com/mnhrdt/imscript

Cool collection of effects. Too bad they are free for non-commercial use only.

That seems a little sketchy for the simpler ones, like this: http://www.fmwconcepts.com/imagemagick/gaussian/index.php

He shows the equivalent IM command lines, and it's 2 lines of text.

No issue with the more complex ones though...it's his work.

He also documented (with example images) and web-published all of them. It's not just two lines of code.

So if I do this in ImageMagick on a commerical project:

  convert $infile \( $infile -blur 0x$width \) -compose minus +swap \ -composite -normalize $tmp0
  composite -blend $mix%x100% $tmp0 $infile -matte $outfile

I now owe Fred something?

So everyone that ever figures out how to do something neat with a free tool and documents it gets to claim the rights to that and ask for money?

Might not even be against this as an idea, but the premise seems shaky to me, I guess

Maybe more to the point is- if you want to ask for money for doing work like this, don't publish the ImageMagick command equivalents? Though I guess that would be hard to avoid other than putting it in some proprietary UI/app or something. Oh well.

No, if you save time reading imagemagick documentation by grabbing an off-the-shelf script, you owe him something

It's on the honor, ethics system. If you did the work, no worries.

> please contact me (Fred Weinhaus) for licensing arrangements

To put it bluntly: pay the dude for the effort.

It’s amazing they are free for non-commercial use. It’s even more awesome they let you pay for a commercial license.

Reply on news.ycombinator.com