Live Computer Vision Coding
livecv.dinusv.com
livecv.dinusv.com
https://www.adaptive-vision.com/en/software/
http://www.adrianboeing.com/improvCV/
https://github.com/utwente/parlevision
I don't use any of these but I see a few of these reinvented every year with not much differentiating them from each other. I wonder why none of them caught on yet?
I think they are useful compared to the tedious workflow I am stuck on (prototype with python notebook to dump visualizations and play with hyperparameters, then port to C++ later). But the fear of getting trapped into some unsupported edge case and having to deal with a messier port usually stops me from changing.
Because the data pipeline is never the hard part in computer vision.
1) making the algorithmic bits in the first place (see opencv, vxl, itk, or insert any other vision library here where many of these bits are made for you)
2) knowing how to achieve the desired result using those bits
Many of these data flow tools show you boxes and arrows, and pretend that connecting bits with arrows instead of function calls is where the magic is.
But if one of those boxes is called, say, "ComputeOpticalFlowBetweenTheseTwoInputImages", how is that any different from calling the ComputeOpticalFlowBetweenTheseTwoInputImages function in the underlying algorithmic engine that they're building on top of?
The new layer doesn't solve either of the two hard problems for you.
Giving a live result is cute, but it's not something that you can't already do with a Python IDE without having to learn some new arbitrary modeling language.
There can be a ton of difference, at least in the theoretical possibility space. But the tools, especially free, open ones have not matured enough to show the difference yet.
To begin with, on a text based environment you may need to look up the library's documentation to figure out the available operations within that. A visual environment can visually show you all the operations. You can say that there are certain environments and editors that can provide autocompletion, especially static typed languages, but there are several limitations even in that: First, the autocompletions almost always doesn't have much context built into them, so they will keep showing the same completions on several contexts and can be overwhelming. So the user need to go back to the documentation to figure out the available operations in that context. A smart, environment (can be either visual or text) can show completions for that context alone. Second, there can be many reactive parameters on a function call, especially if the parameters are long or complex objects that requires extensive studying of the underlying documentation. For e.g., if you are changing a parameter, another parameter's value might depend on what you have given in the previous parameter. A properly constructed visual interface can show and hide or change the structure of the parameters based on the rules and can replace the documentation if properly created, saving the users from frequently referring documentation or causing unnecessary compile and runtime errors. You might think it's a small thing, but if you look closely, this is where most of the time a developer spends, apart from figuring out the sequence of operations to perform. Now a text based environment can do most of these "smart" things, except for showing the sequence flow.
Now for the sequence of operations, a visual or data flow language, if properly implemented can arguably be better than text, because A) It can automatically show the available operations depending on the sequence (or the context). B) More importantly, it can show the big picture at a glance with the visual connections between the sequences. This can be a life changer, especially during debugging and when you have large number of programs or files.
Now in my experience there are no proper visual implementations in public domain that successfully done both A and B, that is reducing the general public's confidence in such a platform. But a proper implementation can make the life of developers significantly better, except a catch - whether the developers need it or not. It's like, It's almost like asking a truck driver, whether he wants an AI assistant or not and he somehow gets that while it makes his life easier but also significantly lower the barrier of entry for truck driving skill :)
ps: I am building a visual environment that can hopefully meet many of things said above..
edit: fixed few typos and added a disclaimer.
I have a flag that dumps out each stage of my CV pipeline when i need to debug something. I imagine this would be useful but its nice to be able to use my actual code base vs a separate system.
I wanted to make a (very) short movie with a similar feel to Waking Life, except instead of rotoscoped images I just wanted to do varying degrees of edge detection on each frame. It came out looking pretty good but it took a lot of tweaking individual numbers in a python script and repeatedly rendering the same clip before I could even get a set of choices to decide between.
Having something like this with me when I was filming it would have made a huge difference!
It's great to start out with and play around with but eventually you're going to want the GUI to get out of your way and just let you edit the code directly. At which point something like shadertoy is (IMO) the best of both worlds.
OpenCV has a dnn[1] module that can import models from other frameworks like Caffe, TF and DarkNet (YOLO's native framework) into OpenCV's own DNN object model, and run inference on them. Both Yolo and R-CNN families have been implemented as examples[2].
What's missing is that OpenCV does not abstract all these behind a single object detection interface if that's what the LiveCV side expects. So such a plugin would have to define such an interface itself and provide an adapter per model.
[1]: https://github.com/opencv/opencv/tree/master/modules/dnn/src
[2]: https://github.com/opencv/opencv/tree/master/samples/dnn