Comment by slacka | Hacker News Reader

Comment by slacka | Hacker News Reader

> "For the programmer, that means we must grapple with problems such as concurrency, locking, asynchronicity,.."

From my perspective, this is the problem of this decade in HW and SW engineering. Around 2007, Intel hit the wall with single threaded CPU scaling. It has gone from doubling every 2 years to a few % increase per year. We at the beginning of this paradigm shift to massively multi-core CPUs. Both the tools and the theory are still in their infancy. In HW there are many promising advances being explored, such as GPUs, Intel Phi, new FPGA, and projects like Parallella.

The software side also requires new tools to drive these new technologies. I think traditional threading will be viewed as a stopgap hack and will be replaced by some form of functional, flow-Based, and/or reactive programming models.

A few years ago, I had to help our EEs write some testing software in LabVIEW. I was blow away by how elegantly it solves concurrency and fault tolerance to bad input data. It took no extra design to utilize multiprocessing and multithreading hardware. The synchronous model in our program eliminated the issue of deadlock and race conditions problems that come up when using asynchronous threads.

Another project with great potential is NoFlo. What SW solutions have others run into in this field?

I've been using multi-media 'coordination languages' like Max/MSP/Jitter, Pure Data, SuperCollider, Touch Designer, and vvvv for years after I got hooked on the model from my first synthesizer (the Nord Micro Modular). When I had to learn traditional programming in school (Java at first), it felt hopelessly barbaric to me. Imperative text-based languages are incredibly time consuming, error-prone, and difficult to use in comparison.

Don't get me wrong, dataflow programming has its own gotchas– explicitly guaranteeing proper order can be tricky – and it's very hard for people from a traditional background to switch over (they're always reaching for the for-loop that isn't there), but it's quite clear to me that I can throw together a complex multimedia system with, say, camera input, some computer vision, effects, video playback, etc. in, say, 3 hours whereas it would take somebody starting from scratch in any other language at least twice that time if not 10 times[1]. Moreover, after they've finished, they enter a much longer period of intensive debugging, whereas dataflow systems I've built have been much closer to bug-free on the first draft.

Whenever I introduce this stuff to engineers, it blows their minds. I'm beginning to feel like it's a matter of serious importance to spread the word far and wide about alternative programming paradigms that make many of the problems of this decade magically disappear and make programming more efficient and easier to learn.

Shoot me an email if you're interested in more thoughts / pointers on this topic – skiptracer at gmail.

[1] Somebody might be able to argue that they could be just as fast with Processing or openFrameworks, but anecdotally developing in those environments is still 2x slower and bug fixing goes on forever.

Just when you think you've found all the environments for creative coding, you discover new ones and wonder who are these people outside of my clique who use them?

Those are just some of the big ones! The list goes on: Polycode, Field, Overtone, ChucK, LuaAV, Plask, Lubyk, and many more.

Polycode looks remarkable! It's like Unity3D for creative coders.

There's something I've ignored here, which is that most of these systems are at a higher level of abstraction than text-based languages. Modules (called "objects") in Max, for instance, are written in C. Max is more analogous to Unix than an actual programming language (by admission of its author).

A question for me is why we write things like web servers in a programming language at all. After using dataflow systems, I believe there are much better abstractions possible than Rails-style web frameworks.

Same for most user interfaces. I am writing client side iOS apps and frankly doing most of that stuff in Obj-C, or any text based language - seems to make very little sense. It's a tool that seems ill suited to the task.

These ideas are not new. And _general_ click and drag programming has been "just around the corner" since GUIs became popular.

Now in specific domain, industrial & control system, signal processing, even designing GUIs themselves, click and drag has worked. It lets non-programmers but subject matter experts get stuff done.

But for generic stuff it hasn't worked too well. UML was one such push. Managers were just going to draw a diagram on the screen, click, "Generate Code" and bam, no need for silly code monkeys anymore. Well it hasn't quite worked that way. Now the "Generate Code" button just does an SMTP send to a .ch or .in domain some place.

There is another such system. Quite exotic called DRAKON.

https://en.wikipedia.org/wiki/DRAKON

It comes the Soviet space shuttle program. Yes they had a successful space shuttle once that few unmanned and even landed itself (before US space shuttles could even do that). Some of the people working on that wanted to bring programming down to non-programming engineers. And created that system. There was some interested in it lately as well. But can't say it exactly took over the world.

In my casual experience with such systems, they are extremely limited compared to something like Max, which is in turn extremely limited and badly designed.

Max is not just 'click and drag' – you can do quite complex configuration, cross-modulation of signals and parameters, etc. plus general purpose programming. You can also script things with JavaScript or Lua and write new routines ("objects") in C, Java, languages that compile to Java, and JavaScript.

They key is that the dataflow environment specifies a strict interface between "objects" and an overall execution pattern, but you can dig in to the stuff you're clicking on and change/optimize the underlying code.

Recently Cycling '74 (who make Max) added a new lower-level dataflow language that maps to C code or alternately GLSL shaders, which makes it super easy for people who don't know how to program to write efficient, auto-optimized audio and graphics routines. Another system along those lines is Faust: http://faust.grame.fr/.

For a more open-ended system that combines dataflow ideas with traditional programming concepts, a time-line editor, and in-text GUI controls, check out Field: http://www.openendedgroup.com/field/.

DRAKON looks like a naive low-level attempt at sticking imperative routines into flowcharts. Diagrammed UML systems sound awful – you probably lose all the power of object oriented programming by transposing it into a GUI. Signal chain diagram languages like MATLAB's SIMULINK[1], Analog Devices' Signal Chain Designer[2], or Cypress' PSoC Designer are usually very basic and nothing like what a proper dataflow system could be.

[1] http://www.mathworks.com/products/simulink/ [2] http://www.analog.com/en/content/Signal_Chain_Designer/fca.h... [3] http://www.cypress.com/?id=2492&source=productshome

Don't get me wrong, dataflow programming has its own gotchas– explicitly guaranteeing proper order can be tricky – and it's very hard for people from a traditional background to switch over

Author of NoFlo here. You're right in that Flow-Based Programming requires a bit of a context switch when compared to traditional programming.

In general things are stateless, and information sharing happens only through information packets (which in NoFlo can be anything you can use as a value in JS... objects, arrays, functions, numbers, etc).

For synchronization we have quite a lot of tools in NoFlo. Information packets can be annotated with metadata ("groups"), which can be used to merge data together, for sorting packets, to synchronize flows, and for routing things to different parts of the graph.

I just launched the new NoFlo website at http://noflojs.org yesterday. Now the next big task is to write more documentation and examples there on how to do different things. And of course there is Paul Morrison's book on FBP that talks a lot about things like synchronization, looping, and deadlocks.

In general FBP does provide a lot more power to the developer. Being able to see visually where and how things in your software are connected makes it a lot easier to understand the system, and to find potential flaws in your logic.

Listen to this guy. I've know about FBP for awhile, but haven't delved into it beyond hanging out on the list briefly[1]. Seems like there are many great concepts there that go beyond the stuff I use and I bet there are many more to discover in that space, if only more people were doing research in the area.

Do you know of anybody using NoFlo in production? Would be good to put that on your site if you do.

[1] https://groups.google.com/forum/#!forum/flow-based-programmi...

You might want to look at rx reactive for dotnet/mono. I have been using rx in combination with f# for a while and really like it so far. http://msdn.microsoft.com/en-us/data/gg577609.aspx I do not understand why MSFT pronounces the linq/monad aspect of rx so much and dont advertise it's push based dataflow aspect. People would have a much easier time getting additional information and perspectives.

Good information on FBP can be found here: http://www.jpaulmorrison.com/fbp/

It's also interesting to see the relationship between process calculi based dataflow and actor model based agent implementations: http://en.wikipedia.org/wiki/Actor_model_and_process_calculi I do hope that new developments in pull based FRP and push based implementations like rx lead to some elegant push/pull combination sometime.

Alternative programming paradigms - what the original article _should have been_. Absolutely exciting and needed.

Do you have examples of products you have built with one of these multi media coordination languages?

I am familiar with flow based and rule based programming, but in my experience these were very high level abstraction systems which were then used to solve specialized problems they were well suited for. While you could use them for any kind of programming - they were turing complete - it wasn't fun, or easy, or made any sense.

I've used (and continue to use) them exclusively for one-off interactive installations and research prototypes (I don't work in the 'commercial' sector), but I believe a number of people have apps made with Max in the App Store. Max is not purpose built for creating applications, but you can bundle a "standalone" which comes with your 'patch' and a copy of the 'interpreter'.

http://cycling74.com/2012/04/19/get-your-max-standalone-on-a...

Indeed they are typically aimed at specific domains, but I think this is entirely natural for something that's a layer of abstraction above a high-level programming language.

One issue with the systems I'm talking about specifically is that they do run in an environment that is a quite heavy app in itself, so building an application or using the environment as part of another application does not work well – you end up with a lot of extra bulk. In order for this model to take off for consumer apps, you would need OS-level support for apps to make use of a single interpreter instance running in the background, or for the interpreters to be heavily optimized.

But that's not the point – the point is that there are other programming paradigms than interpreted vs. compiled, static vs. dynamic, imperative vs. functional text-based programming languages. The systems I mentioned are for realtime multi-media installations, performances, music-making, etc., but similar paradigms could be useful in other domains.

I've built a number of audio toys in Max, eg a basic Granular Synth and a Sampler. The build process was very different to imperative programming, took a while to wrap my head around. Very fun to play with though.

Also, the audio devices for Ableton Live were prototyped in Max/MSP: http://www.webcitation.org/5uKcsulCc

I did some work in Max4live two years ago and while I've been into dataflow programming (visual and textual) for a number of years previous, this really changed my perspective of what I want from a language. Max/MSP-style languages (I've played with Synthmaker and a few others too) have a few major advantages over traditional text-based languages, especially for exploratory programming or prototyping (and the main problem I see cited for these languages is that code quickly devolves into a mess of lines - literal spaghetti code - but I found that with proper software engineering principles this hasn't been a problem for me, just most users are not trained software engineers). One thing that really jumped out at me is that how not having to name things until you're good and ready (because you can simply connect stuff visually without naming or labeling) means you can focus on designing the code and experimenting rather than trying to come up with names for things as you are forced to do in most text-based languages. Visual debugging in Max/MSP was also quite pleasant.

Can't agree enough. As someone who currently tries to take his first steps in serious web frameworks like rails, but has much more experience in Max/MSP/Jitter and FileMaker, I keep wondering why there isn't a more visual approach to traditional programming.

I realize this isn't an easy problem to deal with but certainly there has to be a way to take the advantages of creative development environments like Max to traditional programming - e.g. a GUI for ruby.

Focusing on the big picture and larger concepts of programming instead of getting lost in syntax would be very welcome for beginners like me.

A few years ago, I had to help our EEs write some testing software in LabVIEW. I was blow away by how elegantly it solves concurrency and fault tolerance to bad input data.

Could you describe this in more detail? I think it would be interesting. How does LabVIEW do those things and why did you find it elegant?

I honestly had to use LabVIEW to generate a heart pulse(simulation for a pacemaker). God I hated it so much, so much so as to make one my passwords "labviewsux". I wish it would die, but that's just my opinion.

Eh, not much love lost for LabVIEW here as well. It put me off visual programming for good.

Yes, it's true LabVIEW has its own issues. It's a memory hog and the UI is cluttered and doesn't scale well for large projects. I don't think LV will be THE model for future programing, but I do think that its synchronous dataflow programming model could be part of the solutions to the concurrency issue that languages face today in the massively parallel domain.

My experience was very different from yours. I couldn't believe how simple it was to add an additional test in parallel. It was just a copy/paste, update the functions, test parameters, and I/O. If only writing multithreaded code in C++ or Java was that simple. After my experience, I could see how LV could be a good fit for computer vision apps running on highly parallel architectures.

It's too bad there not more modern, open dataflow languages out there. Massively complex modern CPUs wouldn't exist with tools like Verilog. As software grows in complexity, these tools may also offer some solutions on the software side.

You may not like LabVIEW, but it is extremely useful in some areas. For lab scientists who need to iterate quickly between taking useful data and running the experiment, it's nice to just copy and paste instead of naming new variables, etc.

Currently other solutions are either Matlab (which sounds soul-destroying) or Python (which is still nontrivial).

Erlang / Elixir

Definitely. Also, for Scala/Java, there's Akka[1], for Clojure/ClojureScript there's core.async[2], and for Clojure/Java there's Quasar/Pulsar[3]

[1] http://akka.io/

[2] https://github.com/clojure/core.async

[3] http://puniverse.github.io/pulsar/

Also Scala Async[1] modeled after C#'s Async/Await, though still has a few "gotchas" in it.

[1] https://github.com/scala/async

No doubt. BEAM VM is a beautiful thing.

Lightweight processes, per process heap, fault isolation, scheduler implementation, iolists.

Looking at it after years of dealing with other VMs I find myself nodding my head quite bit saying "Yup, they got this right, aha, got this right too, and so on"

Besides Elixir there is also LFE (a lisp flavored Erlang), also JOXA another lisp-like language for BEAM.

Other systems usually copy and get inspired by Erlang/BEAM VM but often they pick the concurrency model. The real money is in fault tolerance. They are tied together somewhat, but they are different.

As distributed systems grow larger and larger, it is nice if they scale or can parallelize their concurrent parts but they are not going to be practical if they fail. And failure modes of distributed systems are magnitudes worse than single node systems.

But it is also hard to sell it. In other words telling developers look this system will isolate fault and you can restart part of it and upgrade it while it runs, is not going to get them inspired unless they are experienced and have been woken up at 3am by a pager when a large system is taken offline because a minor fault wasn't properly isolated. Many are rather more impressed by marketing and small speed benchmarks.

Reply on news.ycombinator.com