MIT creates picture-driven programming for the masses | Hacker News Reader

MIT creates picture-driven programming for the masses

networkworld.com

27 pointsprat10 comments

There's more to this project than is hinted at by the "picture-driven programming" label. This scripting system is driven not just by pictures, but by screenshot snippets of the GUI elements that you want to click on / control. The language uses computer vision techniques to match things on the screen (and I'm guessing to prevent programs from becoming overly brittle if minor visual changes happen to the screen). Check out the video on the project's actual page, it's actually kind of interesting:

http://sikuli.csail.mit.edu/

I read the paper and frankly speaking, I am very skeptical that this is an application for the masses.

I found this piece interesting:

>The Sikuli Search prototype has a database of 102 popular computer books covering various operating systems (e.g., Windows XP, MacOS) and applications (e.g., Photoshop, Office), all represented in PDF4. This database contains more than 50k screenshots. The three-feature indexing scheme is written in C++ to index these screenshots, using SIFT [11] to extract visual features, Tesseract5 for OCR, and Ferret6 for indexing the text surrounding the screenshots. All other server-side functionality, such as accepting queries and formatting search results, is implemented in Ruby on Rails7 with a SQL database. On the client side, the interfaces for searching and annotating screenshots are implemented in Java.

... and the scripting engine is in Python.

Sometimes I wonder if Computer Science wouldn't have been better off with a constantly evolving C rather than all this mess. Every time I build a web application I feel as if I am building Frankenstein's body with different animal parts.

There were two major attempts to bring computers to the masses COBOL and SQL and they both managed to produce the opposite result rather than the masses using them a demand was created for very highly paid Programmer jobs! Electronic Engineers never attempted to teach the masses how to build TV's. In the late 80's I was teaching laymen things like 'How to Use MSDOS' and 'Introduction to Wordperfect' which paid for the rent but overall was a depressing experience. I also taught at the same time 'Pascal' and 'C' to Mechanical Engineers, my conclusion was that some people take to programming - like some people take to maths, they enjoy it and after an initial push they will teach themselves the rest and the rest will never touch programming even if their life depended on it.

In my mind the only successful 'programming for the masses' products are Excel and partly HTML.

I don't believe that COBOL was designed for the masses - after all, this was 1959, when there were so few computers and it was believed by some that ultimately we would only need one! Also, SQL came out of academia and I don't believe that was for the masses either. I would say however that BASIC was intended for the masses, and as such it was successful. A generation in the UK learnt BASIC on the ZX80, ZX81, Spectrum, BBC Micro etc.

Sadly, when CODASYL committee was formed to develop a programming language ideally suited for business data processing needs they were looking at a language that 'normal operators' could learn and that it would be similar to a human language that is why you get Cobol syntax like:

     ADD YEARS TO AGE

A similar motivation was for SQL. We will give you a database and a query language that anyone can use:

    SELECT title FROM Book .....

During the early days of computing the thought prevailed - pretty much like now - that it was a matter of 10 years before computers would be able to talk in natural languages and AI was around the corner.

But, you are right about BASIC that was my first language as well, quickly dropped for Turbo Pascal which was the sort of the in-thing for a while.

Wow. This might be perfect for advanced screen scraping.

"GUI Testing Using Computer Vision"

http://groups.csail.mit.edu/uid/projects/sikuli/sikuli-chi20...

EDIT: It appears it cannot grab the actual text using the Native APIs so you would have to include a pipeline to OCR black-on-white text

There already is a visual scripting environment--its called LabView and its terrible. LabView is used mostly by scientists who want to control their experiments but don't want to learn how to program. It's very effective when dealing with hardware and you just want to get it working. But if it doesn't work, you're screwed because its very difficult to debug.

That's actually the problem I forsee with the visual scripting environment. There is no better single point of instruction than a compiler error or syntax error highlighting. If you try to circumvent that process to make it more "accessible" to the public, you're going to keep your visual programmers in the dark as to what is actually occurring.

Did you even read the article? This isn't a visual scripting environment, it's an intuitive way to use Python to do image processing.

Edit: here's a paper with examples. http://groups.csail.mit.edu/uid/projects/sikuli/sikuli-uist2...

Ah. You're right. I misread it. However, I stand by my points on visual scripting in general. Even if there is a way to automate certain tasks via GUI elements, you are placing a great deal of trust on the system to operate as expected.

I do quite a bit of research in machine learning, and I can say that these systems are never perfect. Building an abstraction on top of an image processing technique is interesting, but I really wonder how a system like that can be debugged if something goes wrong.

For the record (though as other commenters have pointed out, this rant on LabView is off-topic): LabView is a very successful tool that is used by many programmers in various fields, not just instrumentation. LabView does have compiler errors and syntax error highlighting; they don't look the same as in a text-focused language, but since G is a programming language, it has its own syntax and errors can occur.

Programming languages are an abstraction. G is just another abstraction. I've used BASIC and C compilers with terribly cryptic, useless errors too; the errors had more to do with flaws in the compiler and IDE than the language itself. If errors in LabView are cryptic or obtuse, I would assume something similar.

I thought this might be similar to the incredible Baltie programming language: http://en.wikipedia.org/wiki/File:Baltik3-screenshot1.PNG

(it isn't)

Reply on news.ycombinator.com