NoFlo: Help Everyone To Understand And Visualize Code
techcrunch.com
techcrunch.com
Most code, when visualized in sequence or relational form, is generally incomprehensible even to developers until they've spent time understanding the intent and history of the domain. I can't convince most senior developers to learn to read and use the simplest of UML sequence diagrams. How are non-technical stakeholders supposed to be able to grok this?
Code is complex in the best of times, early in development. Legacy code that's actually used for business in the real world, even when visualized and distilled down to its simplest form, is generally a labyrinth of polymorphism, duplication, object/relational tension, algorithms, background jobs and asynchronous callback madness.
We've seen this kind of thing many, many times before. So often that by the early 2000s when some Golden Visual Environment For Code, the Holy Grail of "drag and drop", would emerge every couple of years or so, we used to roll our eyes and say, "here's another one." I wish I could dig up some of those links, but ALL of the language in this pitch is exactly the kind of thing we've been hearing for decades. And it never pans out.
The closest thing we have is UML. Look, I love UML, and I'm probably one of the last holdouts for it -- I read Martin Fowler's UML Distilled years ago and still use those techniques to this day. But I've never used it to generate code (a la Rational), or to fully diagram an entire system. Even with a human editor reducing the noise and focusing the diagrams on the core workflow, real world systems become extremely complex and very difficult to follow visually unless you have a long history with the system, an understanding of the patterns and architectural principles that went into designing it, and enough technical ability to grok things like inheritance, object/relational concerns, asynchronicity and the like. For these issues, the code isn't the challenge -- it's the concepts themselves, whether expressed visually or directly in code. Any system that grows to have significant business value will a) NOT be able to be maintained purely through visual environment and b) become so complex that the visual representation will be a mass of lines and boxes that, unless edited down and vastly simplified, will be too complex to read even for developers.
The Holy Grail of a visual, drag and drop environment that could create meaningful business value would have manifest years ago if it were possible or even desirable to do so. This latest attempt may end up having some limited utility for simple applications, but I'm beyond skeptical or pessimistic. This seems hopelessly naive to me.
Before Einstein showed up, most experts in the field of physics thought that they solved nearly all the major problems. It saddens me that the nascent field of CS is showing the same close-mindedness. I refuse to accept your claim that we must live with the complexity. My mom struggled with my Win Mobile 2003 smartphone. If Jobs had your attitude, she probably wouldn't be using one today. There are too many unexplored areas of CS for this defeatist attitude toward complexity.
That being said, I actually agree with most of your criticisms of visual languages on todays HW. But I have an open mind. Like what if the problem with visual languages is that they need a 3D display and a minority report/speech input for UI to resolve the spaghetti code readability issue and all your criticisms? Look at leap motion/Siri. These UIs are not that far off. See what I’m getting at?
I’m sure that with new ideas in HW and SW, we will be able to reduce the complexity of parallel programming. And even if Visual programming is a dead end, I’m thrilled to see people continuing to flush our this area. Call back hell is a real and present problem with JS, I welcome creative attempts to solve it like NoFlo.
Neither is the problem with representation. We've had 3D visualization for years -- spaghetti flying at you in perfectly rendered HD 3D is still spaghetti.
The problem is not with the technology, but in the subjective perception of complex domains, and the real world fact that codebases tend to be more unclear than not, acquire massive amounts of technical debt, and even the cleanest codebases, in pure Uncle Bob and GOF-approved architectural clarity, present a cognitive domain that requires a lot of specialization to understand.
It's like saying -- what if the problem with understanding French is that we need better visual representations of French? Cool -- that might help the French and other Francophiles, but it doesn't remove the conceptual barrier and basic training required to understand the underlying language.
Likewise, if I presented you with a perfect 3D model (whatever that means) of my pub/sub solution built with a Reactor Pattern that messages N number of clients and takes asynchronous events from N number of other clients and persists the results of processing those messages in a sorting algorithm to a key-sorted Hash inside a NoSQL store -- you'd still have to have the conceptual vocabulary and experience to have any idea what was going on.
In other words, the issue is not an extrinsic problem that needs innovation, but a subjective problem of conceptualization that floats independent of language and implementation.
I love Steve Jobs' attitude and take inspiration from it -- he put user experience and elegance over the needs of programmers. But let's stay with Jobs for a minute. Because he was focused on user-centric experiences, the tools he left developers were serviceable but mostly afterthoughts. It was up to coders to service tools, not the other way around. XCode is not widely admired as an IDE, Objective-C is a good language but has its warts, and in general Jobs himself didn't see the biggest challenge as technical or code-centric, his innovations were mainly a finely-tuned intuition for the difference between elegant and simplistic, combined with a will, attention to detail and leadership drive to make those intuitions reality. He simplified touch interfaces, and left coders mostly to their own challenges, and I think he was right on with those priorities. In that, he totally "shared my attitude".
I do have an open mind, or I wouldn't still be coding and pushing myself to learn decades after scribbling out my first lines of BASIC on a TRS-80. But I also have enough experience with many, many codebases and have spent years learning techniques for simplifying and expressing complex domain problems. And the class of visual representation of those domains, of which NoFlo is just one more iteration, has very limited utility in that regard.
Is it possible that a better visual vocabulary for code will come along? Sure, why not? But it will be of a completely different class than what we've been presented with so far, and it certainly won't have been invented by IBM in the 70s.
We are failing to communicate here. Did you watch Bret Victor's from yesterday? I’ll do my best to explain. I said I agreed with most of your critiques on visual programming, it's your assertion that we've tried it all and just have to deal with the complexity that I was referring to. Einstein’s refusal to accept status quo, led him to that beautiful, simple equation. I was not saying that visual programming is or is not that solution. I was questioning your dismissal of it at this stage.
My 2nd point about the 3d Minory Report UI was a whimsical example of being open minded towards technology. Which is why I said "Like what if" I was trying to demonstrate a creative solution to the problem by combining it with an outside tech. I doubt it is a viable solution. On second thought, programming with a Leap Motion controller on an Oculus Rift display sounds very cool, but of course google beat me to it in the 2013 I/O.
> It's like saying -- what if the problem with understanding French is that we need better visual representations of French?
Of course, every field takes training, but some representations do have steeper learning curves. I started off learning simplified Chinese. After I changed representations to pinyin, I was able to learn words about 5-10x faster. So I don't see how your analogy holds with natural languages.
> Likewise, if I presented you with a perfect 3D model....
If you describe it in words or show me a flow chart, I will pick up the flow chart much faster. Machine code, not so fast. NoFlo does not limit you to the visual editor. You can also use FBP text language if you desire.
> But I also have enough experience with many, many codebases and have spent years learning techniques for simplifying and expressing complex domain problems. And the class of visual representation of those domains, of which NoFlo is just one more iteration, has very limited utility in that regard.
"The most dangerous thought you can have as a creative person is to think you know what you're doing." -Bret Victor
I don't know how much experience you really have with FBP like NoFlo. I have done some programming in LabVIEW, and I found it much easier to reason about concurrency than thread based languages. You may think it's a waste of time with its "limited utility" but I'm with MZT on this one, "Let a hundred flowers bloom"
Also I'm calling BS on this:
"[...]is a sort of visual mechanism for helping people not necessarily familiar with code, like executives, managers and others[...]be able to understand at a glance what developers are doing"
Bringing executives and managers in the development process is a recipe for disaster, there's a separation of roles for a reason.
There's always chat about programming being stuck in the '80s but I think that with the recent developments and improvements in compilers, programming languages and toolchain the process of programming is anything but stuck in the '80s.
In one sense, yes. Having people who aren't intimately familiar with a code base/data base decreeing how things must be done is... challenging.
On the other hand, executives and managers do tend to take a proprietary interest in how people's time and the organization's money is being spent, and sometimes this does spill over in to technical matters. One way to empirically assess this would be to survey the executive and managerial offices around you, and see whether there are diagrams drawn there by technical staff, perhaps on a whiteboard, or a piece of paper, or in some electronic form. If this is the case, you may already have the disaster you fear... or perhaps it's just the way of the world.
To my thinking, sharper tools to assist in conveying technical points accurately to non-technical people is an underserved and valuable market, even if you get rid of the snake-oil salesmen.
I feel that once we (the programmers and computer scientists) get some more experience, we'll start designing our systems (including programming languages) for such high-level access. Think Star Trek and the 'ambiguous' verbal commands given by characters to the computer to complete a task.
Somewhat tangential to this, I have a similar problem with math education. We don't show students the high-level uses of math and then work into the depths of the concepts and mechanics of producing a result. Instead, we teach them to memorize the mechanics and for many, the understanding never materializes. We expect these students to build their car from scratch.
I tend to agree with this. The whole thing does feel a bit like the "flying car syndrome", in that we had at one point envisioned that by this point in history we would be dictating concepts by thought to machines that would automagically transcribe them into machine code. It's a matter of expectations and projections outpacing reality.
I'm not saying that "this" is the big step forward, but any technology that maps more directly to our mental model or increases the pool of competent people that can interact in it advances the science of computation.
If i were to point to a specific example I'd say Kismet in the Unreal Engine is the best real world implementation I've seen. It redrew the line between artist and technical artist.
(The danger in trying to invent a next-gen non-plaintext programming environment is the siren song of flow charts, mouse-only editors, XML serializations, and other dead ends.)
http://www.kickstarter.com/projects/noflo/noflo-development-...
I'll be happy to answer any questions on NoFlo or the campaign here after I sleep a bit :-)
Just curious, but can you point to any folks who are using NoFlo (not necessarily the new visual editor tool) for ETL? I do a lot of ETL type work, and would love an excuse to begin to use Node/Javascript instead of Python. Python is awesome of course, but there are certainly advantages to Node's event-driven architecture that would help me, since a lot of times I'm processing huge CSV files that can't fit into memory.
I think NoFlo, combined with the new visualization tool, would have a very big advantage in ETL. ETL workflows are one of those things that are so often shared with executive types in a visual format anyway.....
That sounds cool, but I don't get it. I don't see anything that would allow a "doer" to jump in. A "doer" is not going to know what a string or an array is. Am I missing something here?
I agree. They are easy to draw. They are easy to draw because in doing so you eliminate real world situations and just draw some boxes that elide all the real detail of programming. And in doing so you barely express what will be needed to make the program actually work.
Visual programming has been attempted over and over again. It's the future and I suspect it always will be.
In my experience with LabVIEW, I found that the FBP paradigm greatly simplified process of writing highly parallel code. Whether you use the text or visual UI, I’m excited that NoFlo will bring this concept to JS development.
It's interesting, though, that diagrams seem to only really be useful for humans. Maybe that's another form of Turing test: hand a program a high-level diagram and see if the computer can write a program that behaves as the diagram describes. Perhaps this would even be an iterative process, with the computer doing what it's sure about, e.g. defining a type, and then prompting for more input (possibly in the form of a diagram!) when it's unclear. I suppose a way to approach this would first have a known-good solution, like Word-Press, do the diagrams for it, then feed the diagrams into this hypothetical software-writing program and see how much help it needs to recover something like the original WordPress.
There's even another interesting question, which is what impact the programming paradigm has on diagram descriptions of software. You would see differences at both the framework and the language level: a SpringMVC diagram is different than Play, which is different than Rails, which is very different from Noir. And Clojure programs would, generally, tend to be diagrammed very differently from Java or Groovy programs.
They made some brilliant contributions to CS in general, but - as they realized as well - DF is not the holy grail.
Looking at some other flow chart designs that weren't inappropriately inspired by subway maps
http://www.businessprocessincubator.com/media/catalog/produc...
http://www.businessprocessincubator.com/media/catalog/produc...
http://www.file-extensions.org/imgs/app-picture/2731/labview...
http://www.computer-solutions.co.uk/gendev/images/Example_LA...
Seems there were several design ideas they just ditched? Particularly organizing flows around containers that hold smaller pieces. LabView has this functionality and even then you get some flows that are visually useless http://forums.ni.com/attachments/ni/130/3008/1/Run_away_diag... like their text based counterparts. But simpler flows are clearer than NoFlo.
Meemoo also has a confusing interface. They're twisting flow lines to make more of the UI fit on screen. Is there a clear overview where the flow lines are straight?
Quarz Composer and building Facebook in it is a convoluted nightmare. They could have built a Facebook prototype far easier with Flash Builder and a bit of ActionScript. Keyframe animations are best done in an interface that supports them instead of generic connect the dots.
All that said I believe flow based programming has a lot of potential. But to make it viable for flows that are more complex than straight command lines or 20 elements you need visual tools that are more sophisticated than subway maps. Popularizing flow programming would at the very least motivate programmers to build components that work together.
I assert without proof that any programming model that requires me to do layout is one that does not scale and will fail. Any model that does not allow me to suck in and work with 1MM lines of enterprise code will fail. I don't direct this at NoFLo, which I haven't used, it's a more general comment directed to the multiple attempts over the decades for a visual interface. It works great for domain specific, small problems, and falls flat for larger problems and larger code bases. Anyone trying to make a new tool should be aware of the history and difficulties.
When you want to view low level vs high level code, you switch layouts. May have to manually label your components low level vs high level. So you won't see everything at once but noticeably more useful.
Hard to tell how far ahead these guys thought this through. Maybe with community interest and a cool looking product they'd attract enough people to work out these details. But judging by comments and relative lack of interest, won't be holding my breath.
Similar (standardized) notation: http://www.bpmn.org/
Also, blatent self-promotion (sorry) discussing the issues with graphical programming: http://c20g.com/site/blog/entry/the-promise-and-peril-of-bpm
Let's assume for a moment that TC has that bit wrong. In that case, I'm finding the NoFlo tool much less impressive.
{ name: "", inputs : {}, outputs {} }
?
Specialized lib is always better. The best thing about noFlo is visualisation.
a->b
b->c
d->c
{ name: "a", inputs : {}, outputs {"b"} }
{ name: "b", inputs : {"a"}, outputs {"c"} }
{ name: "c", inputs : {"b","d"}, outputs {} }
{ name: "d", inputs : {}, outputs {"c"} }