Hacker News Reader

Comment by henridf | Hacker News Reader

Rather than build your own, did you consider moving to another distributed stream processing platform such as Spark Streaming or Flink? If so, curious to know what disqualified them.

View on HN

Yes, we did survey the solutions out there at the time. At the end of the day the decision was really made on the fact that we knew the domain and problem we were trying to solve with Storm very well. The same topologies we've replaced with Superchief have been running on Storm for years and the business logic of them has not changed that significantly. Given the team's experience, it was easier for us to replace the streaming plumbing than become experts on another abstraction framework.

So this is definitely not a dig at any framework. If we were starting to build stream processing into our infrastructure today we would most likely start on an existing framework. Also, I imagine as our needs grow we will leverage these frameworks in future proof-of-concept streaming projects that may or may not make sense to move to Superchief.

Makes sense. How do you deal with data arriving out-of-order? (assuming your data is timestamped at the source)

We grab a timestamp when data hits our API (largely as close to source as we can get) and use that to order time samples. For time series the ordering requirements aren't as strict as other problem domains, since we can usually go back and overwrite previous time series data points.

Reply on news.ycombinator.com