I'll build this. It sounds like a useful and fun project to build. Will build it using Crystal to be able to ship a single binary with no dependencies. SQLite will probably be enough for this project so it'll ship with it's own DB.
If I would be doing it, I would probably go with python because of the existing ecosystem: NLP, readability, beautifying, image processing or deep learning, … Probably everything is already existing, the goal would be to assemble the pieces like playing lego :-) in a much more complicated way of course.
But I'm not doing it, so your choice is the correct :-) Have a nice time working on this project!
PS: Even the image captioning is almost already implemented, amazing https://github.com/tensorflow/models/tree/master/im2txt