We would never use Spark or Kafka for small data, not like them. Their choices are misguided.
Their data teams are too big, ours are the right size.
Fwiw, I use Kafka regardless of scale. It's the only mainstream system out there that doesn't encourage throwing away your receipts. It's the programming equivalent of "accountants don't use erasers".
Come on dude, tech we can discuss a lot and there is no totally correct objective response. Irresponsible financials are an entirely different beast. A proportionally big data team in a non-data focused / non-tech company is a low-key red flag, and that's what I'm referring to (i.e. >10% of headcount). CFOs don't (and should not talk) in relatives, like "us vs them".
> Fwiw, I use Kafka regardless of scale. It's the only mainstream system out there that doesn't encourage throwing away your receipts. It's the programming equivalent of "accountants don't use erasers".
About Kafka, as far as I've experienced it and also read about, it is mostly safe if you are paying for managed solutions like Confluent or heavily investing on infra for covering for ordering and other issues. But Kafka is as good a technology as it is recognizedly messy, up to a point where I've seen people loving it and then hating it months-years down the line.
I would stick with batch processing if I don't care for receipts. You need to take care for them? Cool, then take charge but if not then don't tackle complexity your business don't/won't need. That's the point, even big banks have survived without taking care about the receipts.