Comment by lidatong | Hacker News Reader

Comment by lidatong | Hacker News Reader

Parent Full thread

I had a few questions:

1. The posts mention Spark, but were you (the Unison team) also inspired by dataflow / stream processing engines (e.g. Flink, Beam, Timely Dataflow etc.) or any other technologies?

2. Relatedly, are there any plans to add a stream processing API? Or perhaps that would belong in a 3rd-party library?

3. Broadly, where do you think (in your opinion) Unison fits in the ecosystem of tooling? Is Unison specialized at solving big data problems that require distributed execution e.g. something I pull out of my toolbox for a problem I'd otherwise solve with Spark? Or is it a truly general-purpose programming language?

4. With the Tree example, you show the "wrong" way and then the "right" way -- one problem I've experienced in the past is how do you as a new user know the right way. For example, I wrap my entire program in IO then .unsafeRunSync it -- technically following the rules, but got nothing out of doing that. Have you thought about how the APIs could guide the user to the idiomatic solution? Perhaps via types?

By the way, I really loved the FP in Scala book, it was my first step into FP -- thank you and Runar for writing it

Hi there! I can't speak to questions 1-3, but with regards to question 4, some of the examples we generated were born from early mistakes I had stumbled into when writing the `Tree` functions, so I have some thoughts. It would be interesting to codify the behavior of evaluating the `Tree` in more types, but looking back, the thing that would have helped me most in avoiding some of the missteps might have been a richer test interpreter, where I could have "mocked" a network of, say, 3 nodes and distributed the `Tree` across them, emitting information about where the data was being moved, and where the various functions were being applied. We had actually briefly thought about writing such an interpreter for the purpose of this article and it's still something I'd like to do at some point.

So what we were going for with the core API is that it should be expressive enough for all kinds of distributed programming tasks, not just Spark-like or batch computing stuff.

We’re planning on doing several of these articles to show different applications. I think some distributed stream processing library would be super cool and there are lots of sources of inspiration for that like you mention. It would be a more interesting translation than the Spark-like stuff though. Likely using the channels ability for message passing.

> By the way, I really loved the FP in Scala book, it was my first step into FP -- thank you and Runar for writing it

That’s great to hear!

Thank you both for the answers!

> the thing that would have helped me most in avoiding some of the missteps might have been a richer test interpreter, where I could have "mocked" a network

This is a really intriguing idea. Debugging distributed can be a huge headache, and I can see a lot of value in having something like that.

> We’re planning on doing several of these articles to show different applications.

Looking forward to future articles and definitely will be following this project!

Reply on news.ycombinator.com