I wanted to play with paper trading strategies. I listen on websocket streams of cryptocurrency and stock ticker stuff [1], and the service listening on that socket just plops it into Kafka, with the partition key being the ticker name.
The reason I bother adding Kafka largely comes down to the ability to use the Kafka Streams API. Kafka Streams allows me to do time windows of different trades, or lets me do a sql-style join across two different streams, or lets me filter out data that I don't think is relevant.
Kafka Streams gives you most of the fun of a map-reduce framework, but without having to administer a map-reduce server; it's even smart enough to handle internal state by creating intermediate topics and/or creating local RocksDB instances. It's pretty cool.
There's also has the advantage of allowing me to set retention, so I don't have to worry about doing any kind of manual cleanup; I only have to worry about having N days of stuff on the server, and it's trivial to change that.
Also, I have a number of topics, each with 32 partitions, meaning that if I do find any kind of strategy that works, I can very easily scale it up without changing any code.
I'm basically using Kafka as a streaming-only database. I think it's neat but I haven't actually found any strategies that make or lose money. It's just been a fun way to play around with different bits of server stuff.
[1] https://docs.alpaca.markets/docs/real-time-stock-pricing-dat...