Announcing Apache Flink 1.0.0
flink.apache.org
flink.apache.org
Precisely how is this possible in a real-world distributed system? At-least-once and at-most-once are possible with different distribution methods, but "exactly-once" is, well, I have yet to see it actually implemented (even theoretically, see the Two Generals thought experiment). Even their checkpointing algorithm has an assumption that the underlying transport is "exactly once" and "FIFO ordered".
If I had to guess, I'd say that what they're really guaranteeing is "at most once" message delivery. Unfortunately, that has a very different meaning from "exactly once".
http://bravenewgeek.com/you-cannot-have-exactly-once-deliver...
EDIT: fyi sewen is one of the authors of the paper and a founder of flink
For interactions (writes) with external systems, you need transactional integration. There is WIP to do this for transactional databases, for non transactional systems, there are cases where it cannot be strictly solved.
EDIT: To clarify a bit more, I'm looking at the "at most once" as once throughout the entire system. Not once per node. Combining a physical at-least-once with a logical "at-most-once" only works at the per-logical node level, not at the system level. Some processing will occur more than once in a system, which could have negative consequences if the processing is not strictly idempotent.
This means if you observe the inner system, you won't be seeing exactly-once semantics but looking at the system from the outside, the semantics would indeed be exactly-once at the target.
What they don't tell you is that in order to achieve "exactly once" delivery you need to have idempotent writes. For example inserting in a database using PKs.
Trying to ensure true exactly-once is a fools errand. In a distributed system it required guarantees (down to the magnetic material level) that are very hard to get right. If any of those guarantees fail, you don't have it. What you have is a close approximation.
Real world applications usually involve a lot more than counting words in memory.
At-least-once is relatively easy. Combine it with idempotent operations and your work is done.
Storm is fairly explicit in documenting what it takes to achieve this, but it's not trivial, and every system involved has to support certain guarantees.
Spark (streaming) made some pretty big claims about exactly-once guarantees, but it turned out that claim was riddled with holes.
In my opinion, "exactly-once" doesn't imply that there are exceptions to that rule.
Guard against dupes and you'll be fine (easier said than done, obviously), but also know the limitations of the systems and frameworks you are working with.
EDIT: Disclaimer: I am an Apache Storm PMC Member
First, it is crucial to distinguish between "exactly-once" semantics with respect to state inside the stream processor (for example an aggregate computed in a window) an exactly-once delivery to external systems. The former is built into Flink, the later is only possible in some cases (transactional systems) and requires extra effort.
Exactly-once for state inside the stream processor is incredibly useful, because it allows you to implement many non-idempotent operations such that the writes to external systems are idempotent: For example, you compute the complex aggregate in the stream processor and only periodically write the result to the external system (overwriting previous values). Now the external system always reflects an aggregate without duplicates.
That is very valuable and only possible if inside the stream processor, you have exactly-once semantics for state. That does imply that the stream processor has a notion of managed state (in Flink for example the Windows, key/value state, and generic checkpointed state).
Disclaimer: I am a Flink committer.
They also compare their guarantees to other streaming systems, and they're not the only one to offer exactly once.
For our current project we evaluated Spark and Storm, and have gone the Storm way. Personally I spent a little time on Flink, but could not convince the management for more serious consideration. Mainly because of the name recognition, Spark and Storm are so popular already. But I am confident Flink will become a great option, especially with hitting 1.0. I felt the community was very active and documentation pretty good.
Here's the fine print (no fault tolerance, etc.): https://github.com/apache/flink/blob/master/flink-contrib/fl...
My bets are on Apache Beam (Google Dataflow) to create that compatibility layer. Haven't used it yet, hoping it works out though!
https://docs.google.com/presentation/d/10vs2PnjynYMtDpwFsqmS...
It support mostly all use cases. Batch jobs. Streaming jobs. Work is being done to reduce the batch time to ms.
Comercial support is very good from the guys at Cloudera and Databricks.
My 2 cents. I hope It helps someone.
I generally can get by with Batch, but streaming makes a lot of our logic easier, and might allow us to handle some new situations. Flink looks pretty good on paper.
I'd definitely recommend including Storm for a POC. For any POC I'd warn that most of these streaming solutions are operationally a bit of a PITA, which tends to distort POC results.
Most I have found are light in actual contrast detail.
Here is for example a video walkthrough by MapR: https://www.mapr.com/blog/apache-spark-vs-apache-flink-white...
Inside a data flow program, the scheduler tries to schedule as local as possible.
For the inputs to a streaming program (for example Kafka partitions), there is currently no locality consideration, but the locality there changes throughout a program lifetime anyways (brokers change leadership and rebalance)
Flink's DataSet API does assign data to tasks after the scheduling. That assignment respects locality, actually. That lazy assignment makes it possible to handle large numbers of small files, for example.
I believe, Flink tried to focus on query language and optimization, as you probably would in a database settings. In contrast, Spark is sometimes described as a batch processing system, which provides a real-time experience by intelligently partitioning the work.
Most of the streaming tech and developments in Flink are very disconnected from that and have little to do with database tech any more, actually.
We also did a podcast about it if you're interested in digging deeper -- http://softwareengineeringdaily.com/2016/02/03/benchmarking-...
[1] http://www.slideshare.net/ssuser6bb12d/a-comparative-perform...
The squirrel mascot is cute, but doesn't reduce well.
It should be illegal to claim nonsense like that without some comparative example vs a c+= zmq/tcp pipeline doing the same thing