We ditched Spark Structured Streaming for Flink for a Kafka consumer, processing 3B events per day. Its been extremely stable so far, and half the cost of the spark cluster