Visual Programming Languages – Snapshots
blog.interfacevision.com
blog.interfacevision.com
Does it make sense to call a programming language "visual"?
What if I represent a C program by a sequence of smells... The translator device reads the text from start to end, converts it to a token sequence, and emits a different smell for each token. Does that mean C is a "smell-based programming language"?
If the tokens get read by a text-to-speech program, does that make C an "audio programming language"?
What about a different visual way to represent C programs (besides the usual text way)? Maybe we replace each token with a colored block, where each token has its own color.
Can't we dispense with the notion of a "visual programming language"? There are programming languages, and for each language, there are many ways to represent its programs.
> Can't we dispense with the notion of a "visual programming language"?
In most cases I agree that label is a misnomer, but I think something like LabView genuinely does fall under the category of a visual programming language.
- most text-based programming languages have no purely graphical counterpart.
- Most purely graphical programming languages have no purely textual counterpart.
- a graphic language is generally meant to abstract at a higher level and by doing so it has bigger trade-offs than a lower-level textual language (that's why most graphical programming languages are specialised and those that are too broad in their applicability domain generally fail to be useful).
Your suggestions kind of miss the point: it's not about the visual representation of basic tokens. If that's your goal, then of course you could represent any token and syntax with colour, sound, smells but it wouldn't solve any problem: you would still have to use the complete original syntax, and your colourful additions would not offer any improvements to basic textual representations of the code.
Graphical programming attempts to solve other types of problems. Generally, they try to hide some complexity behind a particular paradigm. Some are useful to help in learning because the building blocks can look like a concrete representation of something more abstract, like having a kind of C-jaw that represents a loop.
Other systems will help alleviate complexity using domain specific visual languages that would be difficult to represent textually: electronics for instance, or connection between musical equipment, or animations engines, etc.
Bottom line is that yes, they are all languages but while most textual programming languages are versatile and generally allow any level of complexity, graphical programming languages are sometimes better fits when the domain is much narrower.
They also generally operate at a different abstraction level.
Program representations are like city maps. There's only one city, but there are many maps: the subway map, the map of post offices, the map of gas lines, the map of Wi-Fi coverage, and so on.
I will agree that different representations have different capabilities and use cases. However, text is also capable of great abstraction, e.g.
z2 = BesselJ(n, z1)
Do you think, for example, that a (useful) pictorial representation of Haskell code can be derived simply from the language grammar?
Sure, the map is not the territory, but you can also say the language is itself the map, and the territory is something way more abstract, like "computation."
Perhaps you can take the semantics of Haskell and invent a graphical syntax for it, perhaps even strongly inspired by the textual syntax of Haskell. But it wouldn't be straightforward, and any obvious, straightforward attempt would probably be terrible.
In other words, I don't think it's obvious that there is a "deep structure" of programming languages that can be worked on directly in a wide variety of formats. Unless you count the AST as such a structure, in which case graphical programming is reduced to graphical tree manipulation (which admittedly is the path taken by many of these examples).
By the way, the more I think about this, the less I seem to know even what a "language" is.
Is Lisp a language? No, it's a family of languages, but even Scheme is a language known for its minimal grammar and its infinite extensibility. A common critique says that one programmer can't understand another's Lisp code because there's such a great potential for redefinitions and domain-specific syntactic extensions. If Lisp has a slogan, it's "code is data." Is Lisp a meta-language, or even an anti-language?
Like functional programming, one of the key benefits of dataflow programming is that it is inherently parallel. With the coming 16-core CPUs, GPU, and project like Parallella, this becomes even more important.
As Permit points out here (https://news.ycombinator.com/item?id=7274956), a large part of a programmers job is to build a "mental model of their program".
We abstract things and come up with constructs to represent these abstractions to make it easier for us to program a computer: to improve on our mental model.
So, we come up with different programming languages and label them by their approach. This is a functional programming language, this one is declarative, this one is procedural and this one is visual.
My 2 cents.
For analogy, there's only one true physical Earth, but there are many possible maps (representations) of the same area: one showing the temperature at the surface (using color), one showing wind direction at various points, one showing roads, etc. Which map is correct? Which map is best?
What we are doing is saying: "This is an object language, this one is functional, this one is visual."
Again, your problem seems to be more one of not wanting us to call any programming language with any adjectives differentiating it from others ...
When I see repls with sophisticated real-time output representation it's as much visual as dataflow programming by connecting nodes to an output sink.
Brett Victor's work is bringing them even closer since usually dataflow/reactive was used in mathematical settings (time varying scala or scalar arrays... image or sound) but he demonstrated far more the benefits in non linear logic (games).
It's the disconnected, char per char ascii, file saving, build time, run time tradition that creates an artificial segmentation.
ps-edit: of course he's always clearer, that's why we watch his talks.
As long as we accept syntax as a significant part of what defines a programming language, "visual language" is a reasonable concept.
Edit: Reasonable implementations are another matter. :)
If C was represented in an olfactory manner my code would be deemed a bio-hazard.
I have never happened across this bizarre piece of jargon, but I goooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooogled it, cheers.
Yes. But that doesn't say much.
Visual programming languages don't just naively show some token graph. You can get that with a parser and a graph library for any language with half a day's work.
It's about the interaction models they explore, the levels of abstraction, and the tools they offer.
What you say is like getting rid of the notion of writing, since everything that can be spoken can be also written.
>Does it make sense to call a programming language "visual"?
It very much does. For a visual language it's syntax was designed as a set of drawn objects to be manipulated. That probably also influence its very semantics.
Quartz Composer: A Visual Language.
C: Not so much.
>What if I represent a C program by a sequence of smells... The translator device reads the text from start to end, converts it to a token sequence, and emits a different smell for each token. Does that mean C is a "smell-based programming language"?
No, it just means that it also has a smell-based representation (that is impractical and noone uses). On the other hand, visual programming languages were pragmatically designed to work in a visual environment, and enable coding by the manipulation of graphical elements.
>Can't we dispense with the notion of a "visual programming language"? There are programming languages, and for each language, there are many ways to represent its programs.
A visual programming language is not about a mere representation. It's about the core way the syntax is expressed to the programmer.
What you say is like "there are no declarative, logical or functional programming languages because everything is imperative at the assembly level anyway".
Well, the level at abstraction at which the programmer is supposed to work with a language (the language's syntax) is important, even if the code can be represented in a different way. Interfaces (in the UI sense) are important.
I think this analogy works better for the original argument, as pretty much any sentence can be either written or spoken. There's nothing about English restricting it to be only a written or only a spoken language.
Of course, keeping fidelity between both representations can be a challenge. Our design could be improved by being more diligent in sharing the same underlying model for both.
I imagine there are other cases like this, where you have a visual tool for building expressions for some textual language.
(Are there SQL tools like this? Seems likely there would be.)
With that perspective, we can ask what representations of a formal system do we find easy to manipulate in ways that produce results that are significant for us (in a specific problem context).
A visual representation would certainly be a better match for a spatial problem domain (Sutherland's work, for example). Similarly, an audio representation can work well for an audio domain (ex: beardyman's realtime performance system he presents in his ted talk). Likewise, I'm quite sure a system for exploring cuisine art would match a "smell representation".
The notion of a "universal turing machine" guarantees that all of these representations can be mapped to that of a symbol stream. That is a mathematical result, but which representations appeal to humans involved in various kinds of work is not a mathematical question - it is a HCI question.
When you see this movie you will discover the difference between a Visual Programming Language like CPWCT and a text-based programming language like C
(2) When we use a text-based programming language, think of this using ( command-line / terminal ) of the operating system instead of using the GUI to do your tasks.
(3) Designing a visual programming language is an art (more complex than designing a text-based programming language) because the user interface of the visual language is more advanced.
(4) Using a Visual Programming Language, we can see the visual representation text in natural language (English, French, any other language) and we can change the visual representation and we can allow the programmer to change the visual representation (not easy to allow the programmer to change the syntax of a text-based programming language, we can do this, but the programmer will need to recompile the lexical analysis program, and it will be another language that can't recognize programs that are written using the old syntax, while in visual language we can allow this customization and we still open any programs create before or after this customization).
(5) We can add more dimensions to the visual programming language (like the time dimension). this dimension is not known to text-based programming language. you can move forward/backword during application construction process, and you can run programs in the past, you can see a movie of how to create the program step by step
See this movie : http://www.youtube.com/watch?feature=player_embedded&v=wpxbf...
It seems like programmers compensate by creating a mental model of their program as they try to memorize how it will execute. (The comic This is why you shouldn't interrupt a programmer comes to mind: http://heeris.id.au/2013/this-is-why-you-shouldnt-interrupt-...)
I'm actually working on a plugin for Visual Studio roughly modeled off of Code Bubbles to see if it's practical to solve this problem. (http://codeconnect.io) I think text is the best way to express exactly what you want out of a program, but the way we're interacting with that text can be improved.
The problem ends up being that you're trading one form of cognitive load for another and it turns out that given our text-centric view of writing code, locality ends up having a lot of importance (which is lost in the model you're suggesting). There are lots of other issues with it at a practical level too. I've never seen an input model that works well with lots of disconnected little editors, for example. How do you navigate between them? How do you lay them out? What happens when the graph is disjoint? How do I get to some point in the flow of the program? What if I need multiple things that aren't directly related next to each other? How do you even determine relation?
After having implemented this in various forms about 4 times now, I'm fairly convinced this is one of those things that makes good intuitive sense, but bleeds to death from a whole bunch of paper cuts in practice.
The one case where this does seem successful though is in the step-wise debugging case. There's a very definite flow, locality matters a lot less, and debugging is an inherently mouse-oriented task - all the things you need for a zoom-based infinite canvas to work out.
We've solved some of the practical problems you've described. The new C# compiler Roslyn makes determining relation fairly trivial, something that I'm assuming proved difficult in Clojure(script) and Javascript.
Many of the other issues are user interface/user experience problems that are better demonstrated than explained. Unfortunately, our demo isn't quite polished yet. We're showing it off at our school symposium in mid-March, so we should have something to show before then.
Would it be alright if I ran the demo by you when we finished it? Much of our inspiration for the idea came from CodeBubbles, Code Canvas and the original Light Table demo, so it's interesting to hear that you've decided against it.
> The new C# compiler Roslyn makes determining relation fairly trivial, something that I'm assuming proved difficult in Clojure(script) and Javascript.
I had access to the language models then too, but the thing is relation is not always written directly into the code. This was the point I was making about locality. In a world with mutability, you can't really know what matters. We tend to cope with this by putting code that relates near each other (in the same file, in the same parts of a file, etc), but that relation isn't made specific in the call chain. It could be, and usually is, buried under several levels of indirection. The lack of locality really frustrated people, because the mainstream models we have for programming don't directly relate to flow and so it ended up being a square peg in a round whole kind of thing.
> so it's interesting to hear that you've decided against it
I've decided against it for the way we program now... :) Flow as a general concept is a very good one, and anything that can make that explicit would be wonderful. But we can't program with rampant mutation and callbacks and expect it to work out.
Haha yeah I've actually encountered that project. There are few things I've wanted open sourced as badly as Codeporium. A decompiler only gets one so far.. (You seriously come up all the place when it comes to extending the editor: http://social.msdn.microsoft.com/Forums/vstudio/en-US/e536bc... haha)
>I've decided against it for the way we program now... :) Flow as a general concept is a very good one, and anything that can make that explicit would be wonderful. But we can't program with rampant mutation and callbacks and expect it to work out.
Fair enough.
I think this has more to do with our conventional debugger interface being so horrible, that anything is an improvement. When debugging, I want my code ordered temporally in terms of how it is executing. Code bubbles give us that, but a 1D editor solution might still work better if the code can be dynamically reorganized.
> and debugging is an inherently mouse-oriented task - all the things you need for a zoom-based infinite canvas to work out.
It doesn't have to be, that's just what we have to work with at this point.
I'm sure practical visual programming will be a reality some day, but for now writing code is the best we have.
"I'm sure practical visual programming will be a reality some day..."
I don't follow how you get from the first sentence to this conclusion. Seems to me spending a lot of time "arranging cables and boxes" is pretty inherent to the whole visual programming idea.
We could have a bunch of different cool things to use for programming, but with a majority screaming "no, that's not programming" we kept ourselves from some really neat inventions.
I think this has something to do with one of Clarke's Three Laws (http://en.wikipedia.org/wiki/Clarke's_three_laws).
> When a distinguished but elderly scientist states that something is possible, he is almost certainly right. When he states that something is impossible, he is very probably wrong.
I talk and think a lot about democratization of software development.
People I talk with who really want to program but couldn't get past coding really like the idea of visual programming environments where they don't have to deal with the compiler/interpreter.
When I talk with people who are experienced, they feel there is no advantage to programming any other way than writing code.
From my experience, we have not gone too far in improving the development of software for many many decades and Bret Victor does a good job of pointing this out. I think the way we code now, and specifically how we build out software frameworks, is flawed.
We engineer software frameworks that contain both mechanism and business "objects" when software frameworks should only contain mechanism.
Software frameworks should be coded to be used by non-coders: not by other coders.
Really, a domain agnostic visual language will be hard to develop when it is representative of how we code now.
Alice, App Inventor, Blockly, Hopscotch, Intentional Technology, Lamdu, Minibloq, Scheme Bricks (obviously), Scratch.
I've built these sorts of things before to let users define logic through a web interface. I think that's about the extent of their usefulness.
The dataflow languages seem much more interesting.
Even homoiconicity, I don't see why you couldn't write a program to manipulate the blocks (or whatever) on the screen to create a new program.
I don't know if those environments support such construction, but I don't see any reason why it would be particularly difficult to implement.
There is a VPL based on LISP, I forgot the name, but you could definitely say it is LISP then.
The point was that saying these were all like LISP is like saying they are all like machine code.
http://blog.interfacevision.com/assets/img/posts/example_vis...
was 'omg, lisp displayed with TreeView; and let me guess what the serialization format of this is' (probably XML).
- Aardappel (visual functional programming) [1] - Widget Workshop (something like a cross between The Incredible Machine and visual programming) [2]
[1] http://strlen.com/aardappel-language [2] http://www.old-games.com/download/3399/widget-workshop
There were a few things that set this programming language apart from most of visual programming environments. First of all it was a proper programming language and not a toy project. The graph structure visualized recursion and re-use of variables very well. The code could be evaluated and edited at the same time by collapsing parts of the graph. You could collapse every iteration of recursion in the graph.
The most striking difference with normal programming languages was that all names were purely optional documentation.
Too bad I can't remember the name of this project. I remember seeing a presentation video of this project which got me really excited. It was a language by computer scientists, for computer scientists and not something done for kids or designers to get their first baby steps in programming.
Out of the ones on the list, AppInventor/Blockly/Scratch is definitely my favorite. There are different puzzle pieces for different syntactic structures, statements and expressions in particular. The puzzle pieces are colored by the type of the expression they represent. This makes it easy and obvious which piece fits where.
http://www.digitalmediaacademy.org/wp-content/uploads/2009/0...
The rows are events and the columns are objects; you fill in a cell with how the object should react to the event.
- Windows Workflow Foundation (used for building .Net based service flows) [1]
- ArcGIS Model Builder (used for geospatial processing automation) [2]
- Sextante (open source geospatial processing that plugs into QGIS) [3]
- GRASS GIS Modeler (open source geospatial processing that plugs into GRASS GIS) [4]
[1] http://msdn.microsoft.com/en-us/vstudio/jj684582.aspx
[2] http://resources.arcgis.com/en/help/main/10.1/index.html#//0...
[4] http://grass.osgeo.org/grass64/manuals/wxGUI.Modeler.html
Specifically: SCADE, AudioMulch, [BLOK], Shake, Max/MSP and Epecco. There are also some programs with visual programming elements: Voreen, Blender and ANKHOR FlowSheet.
Links/screenshots in the stackoverflow answer.
[1] http://ocw.mit.edu/courses/electrical-engineering-and-comput...
One that's missing, though, is Max/MSP, a visual programming environment focused around music & other media. It's used heavily by many experimental musicians and artists.
They don't play well with version control and are a nightmare to debug. The developer will also quickly hit a point where the visual nature of the tool becomes more of a hindrance than a help, when they're left drowning in a tangled mass of pipes and boxes.
I would really love to see a fresh take on this idea and an attempt to address some of the above problems.
- Houdini (by far the most applicable)
- Maya (2012+ offers a 'node editor' but has always operated on a DAG.)
- Katana (not widely used, but an amazing piece of software)
- Flame and Smoke (similar to Nuke)
Could maybe add Softimage ICE to that list.
Also check out Grasshopper3D for procedural modelling.
Blender has a regular flowgraph for it's compositor and shading networks (and soon for particles, I think). Plus there's the logic bricks thing for it's game engine.
It has Morphic with a snapshot of Squeak which is fair enough but it'd be nice to also include Self[1] which is where Morphic came from and which has I think a nicer implementation[2] of visual programming.
[1]: http://selflanguage.org/ [2]: http://www.youtube.com/watch?v=NYEKO7JcmLE
Smalltalk has aged much better IMO regardless of it's relative market irrelevance.
Based on what is in this list, though, it would seem to me that e.g. the combination of "Objective-C and InterfaceBuilder" could also be added to the list, which feels silly.
* Structured editing environments beginning with the Cornell Program Synthesizer [1] and including environments like Alice Pascal [2], which introduced code completion (I think). You could even include Scratch in this area, though it might be more about rules.
* Data flow languages as you mentioned
* Direct manipulation environments (e.g. Morphic)
* Spreadsheet environments (e.g. Forms/3 [3], Excel, AgentSheets [4])
* By demonstration (e.g. Chimera [5])
* Rule-based systems (e.g. KidSym [6], Kodu [7], ToonTalk [8])
* ... some of the more unique systems based on other paradigms ...
[1] http://pages.cs.wisc.edu/~fischer/papers/synthesizer.pdf
[2] http://www.templetons.com/brad/alice.html
[3] http://web.engr.oregonstate.edu/~burnett/Forms3/forms3.html
[4] http://www.agentsheets.com/
[5] http://kurlander.net/DJ/Projects/Chimera/resources.html
[6] http://www.sigchi.org/chi95/proceedings/papers/ac1bdy.htm
For the ones that simply replace text with flow, and aren't aimed at small children, I only have pity.
If you want to see visual programming pick off, try giving it to people who are somewhat analytical but more creative (don't let programmers make your product successful). For example, designers or even artists.
Another good way to evaluate it is to see what a designer can do with a tool like: www.divshot.com
I have been singing this for a long time, but I think success in visual programming (for the web) will come when you can clearly separate the frontend and backend and let the magic happen then (meaning, see the results thereafter).
Writing code itself won't go away, but why write code for mundane tasks anyway? (Why write out the entire table/list display when you can simply take a generic one and modify it with some mod tools).
The thing we all need to do though is realize that writing a lot more code doesn't actually make you more productive (even though you may feel that way).
Also, with such technologies available, are programmers themselves not being luddites?
Think about using Unix. For copying one or two files, or renaming the, use the GUI. Need to do the same with 100 files? I think most competent people would naturally go to bash, and write a for loops, or small script. For these things it is easier. (And I am fairly sure that most desktop environments have a had more time spent on them than most visual programming "languages"). Likewise Visual query builder in SQL can build a basic query fairly quickly and easily, but once you know the syntax well, it is probably a lot more efficient and powerful to use text.
[1] -- http://www.legoeducation.us/eng/characteristics/ProductLine~...
There's also Virtools by Dassault Systèmes. http://images.gamedev.net/features/reviews/virtools/Image1.p...
If someone is in a such project it might be a good idea to show what are the benefits over looking at code beside subjective aesthetics.
For example ladder logic and LabVIEW look like a wiring diagram, which is excellent because their niches are electrical engineering. Minecraft is excellent as a game developing platform because you see the game you are developing right away.
My own project (http://flowlab.io) is a visual programming language for 2d games.
2. Thyrd: Which combines spreadsheet and infinite zooms in a very interesting way. http://thyrd.org/thyrd/. Some screenshots are at http://thyrd.org/thyrd/paper/, but the video works better to understand it.
http://arstechnica.com/apple/2012/05/25-years-of-hypercard-t...
http://openquark.org/
http://en.wikipedia.org/wiki/Quark_Framework
https://github.com/levans/Open-Quark
http://resources.businessobjects.com/labs/cal/gemcutter-techpaper.pdfBut HyperCard wasn't really a visual programming language. Indeed, it wasn't a language at all (the language is called "HyperTalk"). Anyway, HyperCard did allow you to design user interfaces in a slick visual way. But the code is purely text based. You can see example code at the HyperTalk wikipedia page
(www.sikuli.org)
The stupid Discus thing they use does not let me comment(What a terrible UI).
Also, don't forget that most of the complicated-looking ones are showing a large program - the equivalent of hundreds of lines of code. Just like you to see a portion of code on your screen at any given time, to work on these you'd zoom in to the relevant portion.
However, for the ones that try to be domain agnostic, they don't seem to work so well.