Luna: Visual and textual functional programming language
luna-lang.org
luna-lang.org
There is a lot to be gained from structures like this in general programming, but what they are showing has already been done for 20 years. The real test is when you get in to branching, loops, and all sorts of things that aren't straight data flow. Whether people realize it or not, pure data flow is basically a solved problem, even from an interface view, it just isn't evenly distributed.
I think the most important thing here is that although VPL was introduced many years ago (like textual programming), Luna is the first language that allows switching between code and graphs on demand.
> You can observe "branching" and some kind of "folds" in the second example on the website.
That is a static branch hardcoded from the GUI. That isn't branching because nothing is switching the boolean, which comes back to question of how you would switch the boolean in this structure.
> a lot of movie studios around the world are testing it right now because it gives much greater power than Nuke's graphical representation
This is a bold claim that I'm very skeptical of and my initial reaction is that this seems like a bit of a bluff. What studios are using it and what exactly are the benefits over Nuke that you're talking about?
> Luna is the first language that allows switching between code and graphs on demand.
This is far from true of course. Compositing program's file formats are essentially data flow scripting languages and Houdini's shading language allows you to look at the shader text while modifying it through a node graph.
It really does seem cool but your claims are pretty bold. If you can show off a general purpose program written like this, something that would typically be done imperatively, then that would be much more of step forward from what has been done before.
> That is a static branch hardcoded from the GUI. That isn't branching because nothing is switching the boolean, which comes back to question of how you would switch the boolean in this structure.
Of course. Such static, GUI hardcoded switching is a very common one when testing / prototyping solutions, but of course it gets finally parametrized by some runtime variable. The dynamic branching looks very similar, with another input connected to the switch. This additional input has a Boolean type and triggers the switch between using the first and the second input.
> This is a bold claim that I'm very skeptical of and my initial reaction is that this seems like a bit of a bluff. What studios are using it and what exactly are the benefits over Nuke that you're talking about?
I cannot provide any further information regarding Flowbox, but you can freely ask them about it. We can freely discuss Luna though. Why its superior over the graphs representations shipped with Nuke or Houdini? In contrast to those representations Luna is a real programming language, that allows you to process any kind of data, not only images or 3D geometry. It can access all system level properties (like disk or network operations). You can also define your own data structures. Moreover Luna compiles to native code, so there is no runtime overhead that comes from graph interpretation. I think that those are just few of very important differences here.
> This is far from true of course. Compositing program's file formats are essentially data flow scripting languages and Houdini's shading language allows you to look at the shader text while modifying it through a node graph.
I was using Houdini for years and I know it really well. Houdini allows for conversion between VOPs and VEX (graphs and imperative code) but not vice versa. You'll not be able to modify text and get graphical representation again.
But I can grok the "textual code" example almost instantly, so fast that the difference in effort is almost laughable. I find the mental overhead in understanding the diagrams vs. "textual code" (that phrase makes me mentally chuckle each time I read it) is orders of magnitude higher. Sorry, but I simply don't buy what they're selling (and I say this as someone who lives and dies by the whiteboard).
About the only thing out of this that looks actually useful for real-world software engineering is the visual profiling piece. If I were them, I'd probably throw out everything else they are working on and focus on that single piece alone, because that could be pretty useful.
One interesting VPL I ran into is a free fractal editor on iOS which lets one build up visual LOGO execution pipeline(s) of operations.
A number of very large organizations are using it: from the military, to Netflix, HP, NASA, Pharmaceutical companies, and so on.
The major benefit is that it lets process owners/managers, who aren't programmers, see and understand the code/architecture behind their processes, and suggest, or even make, important changes to those processes.
I don't think this tells anything whether someone with the same amount of experience in diagrams might be able to make more of it than you with the equivalent text representation.
Frege's notation is another example, although it was a bit idiosyncratic, but its legacy very much lives in proof theory.
And then there are string diagrams, proof nets, ...
Do you know anybody doing maths who could say "well I've never felt the need to use a figure in my life"?
Yes exactly. And if you look at the common methods for how knowledge is represented, in mathematics or any field really, it's very roughly around 80% knowledge as text and 20% graphs/diagrams/figures.
An image can be extremely helpful and illustrative if appropriate. But almost nowhere do you see people communicate primarily with images.
Text is king. I think if the programming environment doesn't embrace text as the primary input method, then it's not really good as a general-purpose environment; it'll only be successful in niches.
I'm not quite sure how humans could communicate primarily with images, at least until neural interfaces are in widespread use... I would argue that, yes, we communicate primarily with text, because we have the innate skill - speech - to do so efficiently and we have no equivalent skill for images. This doesn't tell anything about how knowledge would be best expressed in theory.
> And if you look at the common methods for how knowledge is represented, in [...] any field, it's very roughly around 80% knowledge as text and 20% graphs/diagrams/figures.
Sources, please. But even assuming this is true, this would leave 20% which are better represented visually. And I believe it's a worthwhile goal to integrate visualisations for those 20%. An integration like this seems possible with Luna's approach of offering both, a text and a visual representation.
> Text is king. I think if the programming environment doesn't embrace text as the primary input method, then it's not really good as a general-purpose environment
I agree with you that it's a good idea to have some textual representation - If for nothing else so you can use existing tools. But that seems to be exactly what Luna is going for, if I understand the site correctly.
I don't see why text has to be the primary input method.
Yeah, I'm withholding judgement on Luna specifically, until we can see it in action. Their model could be a good one, it all depends on whether they treat the text representation as a first class citizen or not.
1) Standard AST - one that is a common and extendible syntax representation. Both the textual code as well as visual graphs are able to translate to and from this AST model. To be most precise - both representations are just some kind plug-ins (with some funny type-level-safety built-in) over this AST representation, so they are both first class citizens and they are completely equal to each other.
2) Core AST - a desugared, type-checked, minimal lambda calculus language, that is used to perform all sort of optimizations before next steps towards generating the target code (assembly / js).
We treat the graphical interface exactly the same way, never favoriting any representation over the other. If you would like to develop another one, like natural language input, you have just to provide methods for translating it to and from AST and our infrastructure delivers everything you need to translate it to and from visual or textual one.
I hope this spots a light over the internal design of Luna :)
It's true that there are some visual notations in mathematics, but they tend to be the exception rather than the rule, and where they are used it's a mixed system most of the time. Frege's notation was half symbolic half visual, and the visual part of it didn't catch on except for the horizontal line. The rest of it got "symbolified". I don't think that's a coincidence. Actually the most commonly used half visual half symbolic notation in mathematics is the commutative diagram. I suspect that over time even the use of those may decrease in favour of symbolic type theory notation.
I do think some sprinkling of visual notation is useful, also in programming, but representing expression trees as dataflow graphs? That has been tried many times and it hasn't been successful, nor is it likely to be in the future. I don't think flat text has any future though. A program has structure and is not best represented as an array of characters. Symbols and nested syntax are here to stay however. Humans just deal better with nesting and symbolic references than with a forest of lines and arrows.
How much experience with visual programming?
This is not very useful and often gives you the worst of both worlds - you gain no useful information about the underlying structure but lose many tools to edit the program efficiently. (Though in highly constrained environments - e.g. high level game scripting - even that can be useful)
On the other hand, there are often cases where some relevant aspect of your program has a structure that could be visualized easily - but not as easily expressed in text. In those cases, I think working with the visualisation can be a lot faster and less error-prone than working with text.
The trivial example is of course GUIs or other things which are visualized in the end anyway. But it is also true for data flow graphs (think Storm or TensorFlow) or networks of things that communicate (Components in an IOT container, services, machines on a LAN, etc).
of course visual programming doesn't magically un-fuck your bad design. If you have a big ball of mud and have a look at the component graph, you'll likely see a tangle of lines that doesn't give you insight. But I think that would be more a sign that you should overthink your architecture, not that visualisations are useless.
Nevertheless if you had 20+ years of experience in programming textual languages than it is no surprise that you understand them better than visual languages. That part of your argument does not seem valid.
What exactly looks difficult to parse about the visual example going:
read -> grade -> transform -> grade -> final touches -> write
To me it's a very obvious image processing pipeline, reading the image, grading (in image processing terms this means color work, e.g. adding a hue to the highlights), doing a transformation (e.g. some rotation, or similar), grading again, and finally writing the result to disk.
If you wonder what "exactly" they are doing, then unless you open each box (like the 2nd "grade" box is open), they're not meant to tell you. Closed, they're meant to give a high level overview of how your functions etc are connected.
Think of it as seeing your code in your programming editor with the functions folded.
The second example is also obvious. There's a selectable (by dropdown) input source, currently set to read posts from Twitter. Posts read are then fed into a (switchable on/off) filter, that either passes them all, or only keeps the one from Europe.
Of course you need to spend a few minutes (or days) to familiarize yourself with the visual language (like you would for new terms if you met a new programming language), e.g. how the toggle icon means a "switchable" option.
My point is - I could either waste time piecing together a visual diagram like that (which is going to be fairly different visually from use case to use case - e.g., image processing vs. text processing, resulting in more mental overhead) - or I could look at code, which is (for me, at least) simply much quicker and easier to understand (in many cases).
Furthermore, these are idealized examples. Start throwing in error handling. Concurrent processes. I predict that diagrams that can fully capture complex systems will lean far more towards "visual overload" than not, and really, how useful is a tool like this at that point?
Maybe something like this could be fine if it's just high-level processes, but it just falls apart the more detailed things get.
My suspicion is that as your model approaches the complexity of real-world systems, the visual language you're creating will be unable to efficiently capture that complexity.
Would love to be proven wrong.
Moreover, nothing prevents you from writing code, as they say Luna maps 1:1 to the code they show next to the diagrams.
But when you want a higher level view of the whole program architecture, code can't show you that -- with code you only get some source file or another, and even that rarely all of it in a single screen. A visual representation can, and it can group larger units in single modules (e.g. your packages).
>My suspicion is that as your model approaches the complexity of real-world systems, the visual language you're creating will be unable to efficiently capture that complexity.
One can even make a trivial visual language of 2-3 node types that is turing-complete and can capture ANY complexity. Luna is, of course, far better than that. As for sufficiency, this depends on the level that one wants to play. You might still write code at the lower function level, but if the language makes it easy to group functionality in larger visual units, it would be great for the larger structure, and also great for configurable parts meant for non-coders.
I don't disagree with you on the first part - that you can make a trivial visual language that can capture any complexity. The challenge is - can a user "visually" parse through the visual representation of that complexity?
I haven't seen any evidence that Luna can effectively do that at scale. Until I do, I remain extremely skeptical.
https://www.aspentech.com/uploadedImages/Community/Blogs/Eng...
That chemical process simulator lets engineers who are bad "text programmers" make complex chemical process simulations by giving them a library of functions that correspond neatly with the set of tools the engineer is used to thinking about. Indeed, other engineers can discern the intent of a flowsheet made in this tool faster than they could if they were looking at the text-code.
I always thought there should be a graphical language to do the high level architecture.. And a text representation to do the details. But it should be the SAME language.
Not like CASE tools where the uml diagrams are different than the code.
'The good thing about bubbles and arrows, as opposed to programs, is that they never crash.' — Bertrand Meyer
Now in visual languages.. The bubbles do crash.
I'd love to learn more.
I have not made an extensive research, but this doesn't sound right to me. The SAS Enterprise Guide version of the SAS language came first (and probably others). In one panel you have the visual version while the other has the traditional SAS text language. There are other environments live Learntomod where one panel shows Javascript while the other has a visual, Scratch-like representation of the same code. Just like Luna, they consist of an IDE coupled with two languages each: the visual and the textual. Why is that they don't make that argument false?
Questions:
* The website states that a compiler and an interpreter will be open-sourced, but this language probably requires some kind of "graph IDE" as well. Will that tool be open-sourced too?
* Are the authors trying to make money from this? How?
* Who is behind this?
Will that distributed runtime be open sourced as well?
The bottom of the page says © New Byte Order 2016. This appears to be their website: http://www.newbyteorder.com/ Not much extra info on there, though.
- the textual language does not require IDE to create binaries, only the compiler that will be open source. The graphical IDE is called NodeLab and it will not be released under Open Source model at the moment (althought we are constantly considering such option). It will be free - that's for sure.
- As a startup we have a plan to make money from making a product. Althought our motivation bases on a strong belief that we can change the way people are able to develop software, program IOT and visually process data, the money is needed to hire people, eat something and live somewhere :) Ok, answering the question in more details - we plan to make money by providing paid support and by selling visual data processing cloud solutions. Luna runs locally or in the cloud. Our editor is able to run in web browser as well. We're working on cloud runtime that allows you to model any data processing graph, that our cloud infrastructure will process in parallel and will return results to you. That are our next steps. Currently we want to release the technologies, build community around it and prove that Luna solves problems in different domains by making interesting use cases.
- We are a startup from Krakow / Poland / Europe. My name is Wojciech Danilo (https://pl.linkedin.com/in/wojciechdanilo) and I'm one of the founders. I've got pretty big background in visual data processing - I was working as senior technical director in various movie studios for several years before starting this project. There is also a VC behind us - InnovationNest also based in Krakow.
Did I answer your questions? If not, I would love to provide more information :)
If the concept allows, as some have mentioned, to let non-programmers make changes, then a fully-integrated test framework is absolutely required.
Other questions come to mind:
- How are global 'variables' (config, settings, etc.) represented? As more lines, or traditional named variables?
- Although the Category Oriented Programming seems compelling, I'm not seeing how this visual world will address concurrency in any meaningful way (maybe I'm missing something) - although the profiler is nice.
- Visual programming is also extremely messy when dealing with exceptions: not sure how that is addressed here - again more lines?
Before long you end up with spaghetti... but perhaps they've solved that via views / layers or something.
I highly applaud this effort, but it needs to address the complexities of software development /on the development side/... or it won't get much use by professional developers.
When we are talking about testing framework, we should ask ourselves - what kind of tests are we talking about. In languges such as Python, Ruby or JavaScript (I don't consider JS to be a language, but this is just a side note) everything should be covered with all kind of tests - unit tests, behaviors testing etc. There are languages like C++ or Java, where some testing is not required, because it is done by the compiler - and I'm talking right now about just type system. Luna incorporates much better type system than C++ or Java. In fact Luna will throw compilation error if you want to access head element of an array if it cannot prove this element would be always there (Luna incorporates a complex type inferencer and dependent type system). You can tweak the compiler and allow for more liberal behavior if you like thought.
To sum this up - I'm not talking that test framework is not necessary. It is, it is curtial and need to be polished to allow a language to be widely used. I wanted to clarify, that a great type syste, like the one in Luna, prevents from making errors so much, that you can write much fewer tests and be sure everything works as it should.
Answering other questions: - there are no global variables in the traditional meaning (in fact there are no global variables in purelly functional languages at all). If you want to have something like a global variable you use monads! "Oh no, monads, I'm running away!" - this is a completely understandable reaction for vast majority of programmers. Because of that Luna incorporates an automatic monad management system. What it does is simple and allows using monads without thinking about it. You just put a "State.run" node and you pass all variables to it, that shoudl be accessible from every palce of its sub netwrok. Then you can put "get" node, that will access these "locally global" variables for you. It's just like a State monad in Haskell (this is explanation for Haskellers), but with visual, easy interface.
And one more thing. Global variables are pure evil. I would call them bilion dollar mistake, such as null pointer references. Monadic states are pure and nice solution for it. (btw. Luna of course doesnt have something like `null`).
- The lines you see represent all data dependencies in a graph. Anything not connected with lines (or monadic lines - like two actions that print something to the screen) can be run side by side. Our compiler understands that and can shedule the nodes to be executed in parallel.
- Exceptions are represented on the bottom of our page. Exceptions in the meaning known from imperative programming are messy, ugly and doesnt fit in the visual environment well. In fact they are hard to deal by developers too - have you ever tracked an exception stack of poorly written code? This is pure nightmare. Luna introduces dataflow exceptions - please look at our website and I will answer any qeuestions that could appear after reading it :)
- We have created a really big graphs with Luna without ending with spagetti - each node is build outo of graph of other nodes. If you've got to many nodes on the stage is justl ike writing a collosal 1k lines long function in textual code. You can then select some nodes and collapse them into one (just like refactoring functions out of such code block).
We are aware of all these things and we do what we can to create a REALLY useful visual programming expirence. We've started this project because nobody ever did it and all the tools currently available (excluding some domain specific ones like Houdini, Labview etc) are just toys and cannot be considered programming tools.
Are you referring to this?
fileAge = open 'age.txt' . to Int
age = age.catch e:
print 'An error occured: ' + e
read 'Provide your age: ' . to IntI don't think that's entirely true. For example there is EMC XProc Designer[1]. XProc is a mature W3C standard with several open-source implementations.
With a proper IDE support it's no harder to write than Lisp. So by that definition
> EMC XProc Designer cannot translate graphical representation to code and vice versa.
the answer is yes, it can.
But the most popular tool in this category probably was Yahoo Pipes. They closed Yahoo Pipes.
How can I get my hands on an interesting sample (perhaps a very basic/simple web server that can serve up static html/images?)
With that said, this is still just a concept until a language actually brings all these features together. Will believe it when I see it.
Excited overuse of the word "incredibly", just like the author of the featured article.
> make it seem like joy to use.
Missing article, just like the author of the featured article. Interesting ;-) (OK, this one happens to a lot of people who come from a Russian background, for example.) If you are involved with this, I'm sure people would be interested to know! Maybe you could answer some questions...
Answering your questions:
a) This language is NOT good for any really low-level stuff like programming realtime hardware systems or low-level processors programming. You should then use pure C or assembler instead. If you are programming other low-level stuff, you could feel comfortable using Luna's textual representation. The visual one gives incredible boost when designing the high-level architecture.
b) At the current stage Luna ships with a very limited standard library, but we will work hard in order to incorporate as many usefull libraries as possible. As you have spotted, Luna is able to directly call Haskell and we also provide foreign C interface, so binding any Haskell library is almost out of the box right now.
c) We are strongly working on this topic. Currently our graphs are able to run in parallel all operations that are not data-dependent in any way. However it does NOT scale well if you're thinking about distributed systems, because it's not easy to determine where to partition your program, so sending data over network will not kill the performance. We will though be working toward addressing it in longer perspective.
Did I answer your question? :)
Just a follow-up question on (c) - if you will allow me please. Are the data structures immutable like Clojure and Erlang, so that they could be distributed eventually, once the optimizations are done (and you are able to package the non-functional code like C into their own partitions for the distribution)? Reason I ask is that the page mentions Category Oriented Programming - which I am not at all familiar with. Regards
c) Yes, Luna supports immutable data structures in the form of composable algebraic data types. Composability means in this context that you can compose the sum and product types directly using Luna's built-in combinators, so you can for example build data types that share some constructor definitions (this is just a generalization of algebraic data types in general. As a side note, the first Luna release lacks some mechanisms to fully handle this abstraction, but this is understandable as this is an alpha release). Luna data types allow for efficient binary serialization, if you mean it by asking about "own partitions for the distribution". The Category Oriented Programming paradigm, on the other hand, is developed by us. It is widely described in the Luna user manual, that for now will be available for alpha testers only, but we will work towards releasing it to the public as fast as possible.
Did I answer your questions?
a) When is the first alpha release planned ?
b) Did you go the Coq/Agda route with dependent types (only total functions can be defined), the Haskell (once the DependentTypes extension lands) route (partial/looping functions can be used in types and may need to be evaluated at compile time, the compiler can loop, there is no associated coherent logic), or the Idris route (only total functions can be used dependently) ?
c) How extensible is the Standard AST ? (e.g. would it be possible to define something like the Coq tactic interface from within the language, or through a compiler plugin ?)
a) We will release it to the first group of alpha testers in couple of weeks from now. This will be couple of weeks, not couple of months.
b) We're going the Haskell route - we allow for non-total functions, and we will extend Luna's type-checker to understand as much as possible. In fact we treat type level very similar to the value-level, very similar to the type of types - level, etc, so there could be functions evaluated at compile-time and they could cause infinite loop. Please note that although we design Luna in order to be fully dependent type language, the compiler just hit the alpha stage and there is much more work to be done in this field.
c) The standard AST is really very extensible. In fact every term that belongs to AST is just different data type. We've written a library in Haskell that allows handling open sum types using all the type-level machinery to help it be bullet-proof. Luna will incorporate a library that allows you to change and manipulate the AST from within a language - something like TemplateHaskell, but without the stage restriction known from GHC. This is not available at the moment though. I doubt though you will ever need to extend the AST using a plugin, because it allows to express really wide logic, however I'm very interesting what idea do you have in your mind here! :)
Anyway, looks interesting. I'll be keeping an eye on it, even if the whole visual part turns out to be a steaming pile, the text-based language looks quite promising.
It's really great that you're addressing this. Thank you! ;-)
Edit: Not to be confused with another Luna Language: https://github.com/luna/luna
The concepts in Luna were influenced mainly by Haskell, Idris, Erlang and other functional languages.
Doesn't matter.
This doesn't look any different. Someone already said it, but the fundamental problem with these are that it's extremely hard to build big and complex software without making it look like a huge mess. In a normal programming you can always call any part of code and make any number of instances and put them almost anywhere you want with very simple, logical and linear code which is executed in a normal standard prodecure (= line after another). With this almost everytime you make some changes you'd have to rethink the whole program flow and move all the components/blocks to different places and there are millions of lines going all around the place.
1/10, will fail.
In my view, text based languages could be this way too, except that the habits of making code at least marginally readable (e.g., indentation) and modular (subroutines, local variables, etc.) are drilled into everybody who learns programming, or learned through "the apprenticeship of observation."
Also, those habits require only a bare minimum of physical effort, such as reaching for the tab button or enclosing some code in a function definition.
I think a problem with graphical programming is the sheer physical effort that it requires, encouraging sloppy work. Maintaining readable code as it grows in complexity is a major chore, involving physical and not just mental, effort. In practice it doesn't happen.
What I'm trying to say is that it's very likely the user doesn't even know what "functional" means. They just hear some keyword they don't understand and dismiss anything that follows. Because when you think about it, it doesn't really make any sense that a programmer could be a programmer if they couldn't look up things they don't understand.
I don't want to learn something else ;)
Maybe programming isn't for you then.
Personally I find those that equate learning new things in CS with learning new programming paradigms (while common on HN/Reddit) to have a very skewed understanding of what computer science is about.
1) Unfortunetally the majority of schools / universities teaches imperative thinking and imperative languages only, but this situation appears to slowly change.
2) Functional (especialy purely functional) languages provide completely different syntax than the "standard one" we know from Python- or Java- like languages. Lets consider Haskell or Erlang for example. If you know any imperative language, reading Python or Java is just "slightly different", but understandable. You cannot understand Haskell, Erlang or Lisp code though. This was a very important factor for us when designing Luna - as you can see from examples on the website - the textual code is widely understandable, keeping at the same time all the design decisions derived from the fact that Luna is purely functional programming language. I think that this aspect of Luna will be very important for new-commers from the imperative world.