Wallaroo provides the infrastructure plumbing to write resilient, scalable applications. You write your code to our scale-independent APIs (currently Python 2.7 is supported. Go and Python 3 coming later this year). Wallaroo hands laying out your data and computations across X number of processes which can change as the application is running. We have built in message replay, are working on completing exactly-once message processing (at-least-once delivery with deduplication). Wallaroo handles the infrastructure concerns so application programmers can worry about the business logic.
"the machine learning scenario" is rather broad so I can't answer that specifically. We just starting working with a company that is interested in the ability to take the Python their data scientists are writing on their laptops and move it into production with no logic changes. Currently they end up rewriting in a JVM based language and often have to switch machine learning libraries.
There's a 15 minute video I did about scale-independence that includes code and a demo that you might be interested in:
The vision here is to make it much easier to build and scale distributed data applications in general, not just event-by-event. We are in the process of putting together functionality we need to support microbatch type workloads and jobs that have a beginning and end.
re: Too general/too specific. my personal email is in my profile. i'd love to hear from you on what was too general and what was too specific. We did our best with explaining but we knew that we would still end up being off the mark in some cases. If you could spare the time to drop me some notes, I'd appreciate it.
Sorry if that answer is a little rambling. We had a number of long hours while cleaning everything up for the first open source release and I'm exhausted.
2. How does Wallaroo differ from just running bunch of AMQP clients for data processing? And I ask this as somebody who's in the middle of writing his own stream processing engine.
And a few comments:
- Installation procedure is so unnecessarily long. It would be much shorter if you provided source package and not require anything from outside of the distribution.
- Installation procedure involves downloading Pony compiler from non-canonical source. It's a warning of sorts. I don't know if the compiler source was changed from the upstream, so I'm wary about Wallaroo's compilation with official Pony compiler.
- Installation procedure only ever mentions Ubuntu. You do know that people run other distributions, right? Does Wallaroo even compile under Debian or CentOS, or was it written for Ubuntu specifically?
- Requiring Docker to run examples is bad. Your software should be easy to run without garbage from containers.
Thanks for your comments.
- We are pairing down the installation procedure. We couldn't do everything we wanted before we did our first release.
- The compiler was changed from upstream. We are in the process of getting back onto mainline ponyc. That should be happening in the next couple weeks at which point, we will be taking the first step in pairing down the instructions. When we were first developing Wallaroo, we wanted to be able to rapidly evolve the Pony runtime as well as Wallaroo itself. To do that, we needed to fork the project. We've gotten almost all of our changes back into the mainline ponyc.
- It works on other distributions. We currently only provide instructions for Trusty and Xenial because as you said, it's a somewhat involved process at the moment that we are working on simplifying.
- We went back and forth on if we should use docker for the UI. Its written in Elixir and requires that Erlang be installed. After working with some clients, we decided it was easier to provide a docker container.
The Python version embeds a Python 2.7 interpreter inside a running application. We have C code that acts as an interface between our Pony runtime and the developer's Python code, calling the required framework functions at the correct time.
The event-by-event features in Wallaroo are somewhat analagous to Storm but we had management of in-memory state that allows you to grow the size of the cluster and have the application state redistributed across the cluster. This can be done while an applicaiton is running. In the case of Storm, this would be like:
1- keeping your application state in objects in your bolts (this helps with performance compared to an external store) 2- having a means to add more workers to the running storm topology and have that in bolt state redistributed across the new number of machines.
The 15-minute "scale independence" video shows this in action: https://vimeo.com/234753585
In many ways, Wallaroo looks a lot like the systems that Pat Helland describes in "Beyond Distributed Transactions": http://queue.acm.org/detail.cfm?id=3025012
One of our engineers is giving a talk at Velocity conf NY next week. He's going to be discussing Pat's paper and how Wallaroo exhibits a number of characteristics he describes.
https://www.monkeysnatchbanana.com/2016/11/22/codemeshio-how...
So I think you could get out more from your demonstration and place it before the API usage tutorial part.
Right now the demonstration feels a bit hasty and nervous. It would be nice to have more detail on whats going on in the demo.
Do we think asking people to watch a 30 minute intro video will cause far more people to drop off than a 15 minute video? If yes, can we convey a lot of the value of Wallaroo in 15 minutes and get them interested in learning more?
We decided the answer to both questions was yes. I'm planning on doing a couple longer intro videos for those with an appetite to learn more at a slightly more relaxed and less hurried pace.
My question to folks would be, what's the longest you'd be interested in watching? I have enough to cover that I could do:
30 minute, 60 minute or 90 minute videos, each providing more depth. How much time are folks generally interested in investing in videos of this sort?