Now I'm instead trapped with Go, which lacks a REPL, and leaves a lot to wish in terms of metaprogramming capabilities needed to create a nice programmatic API.
The lack of the mentioned features is my biggest concern with Julia.
For example one could use a monadic style data structure such .NET's reactive Observable (which has an analog in many different languages) and allow you to compose a stream like definition independent of how it runs. You can then feed this definition (Observable) to an evaluator which could run it on Go using channels or it could run it on Go using a cluster of machines with ZeroMQ.
I think the selling point to your library is that it compiles to native code but my question is with out changing code can I switch to running it on a cluster instead of local (something I could do with a language with an abstract form of concurrency using monads or Observables or even just using the Actor model)? It looks like I can right?
That is, for multi-node jobs. As long as you can stay within the 16-32 or so cores on a typical HPC node (in our cluster at least), scipipe should be great for that. I think some means of simple resource management (to not overstress these 16-32 cores) is needed, but that can be done in a simple way by e.g. using a central go-routine that lends out "the right to use a cpu core" on demand.
Thanks for interesting feedback. I will think about this!
In either case I couldn't get my head around how to implement this.
In Go, the implementation is conceptually extremely simple (although the code might not always be the most readable).
From what I can see in the docs I'm getting afraid Dask does the same mistake as so many other recent tools: Allow only task dependencies, while what is needed in general scientific workflows is data dependencies (connect outports and inports of process). I have explained this difference in a blog post earlier: http://bionics.it/posts/workflows-dataflow-not-task-deps
(UPDATE: in all fairness, they seem to be doing something in-between, a little like Snakemake, in that they allow to specify data inputs based on a naming scheme. What we want is a totally naming scheme independent declarative way of explicitly connecting one output to one input, as that is the most generic and pluggable way you could do it.)
If they allow true data dependencies though, that would be very interesting.
How about this? https://github.com/shashi/ComputeFramework.jl
But I suspect it has the same problem.
Edit: There is also this https://github.com/JuliaDB/DataStreams.jl
http://docs.julialang.org/en/release-0.4/manual/parallel-com...
Which to me as Limbo programmer, look like channels, even across machines.
This is common practice in bioinformatics already, because of the large plethora of tools which would be too much to rewrite for any particular language.
Then, Go will probably be OK for more mundane tasks such as data pre-processing, filtering, etc etc.
Otherwise, there are in fact some scientific Go libraries already, including BioGo [1] and GoChem [2].
This would mean that we could automatically create true struct-based components with channel fields from the shell like syntax used in the examples in the README (like "echo > {o:foo}" to write to an out-port named "foo"), so that connecting an out-port to an in-port would go like:
Process2.InFoo = Process1.OutFoo
This is not possible with Go's reflection though, so right now, based on these shell like patterns, we can only populate maps (InPorts and OutPorts) of the process, such that the above code example becomes:
Process2.InPorts["foo"] = Process1.OutPorts["foo"]
I don't understand why you'd want to do that. That sounds like an architecture you are excited about, not a problem you are trying to solve. Can you give me some context?