Thanks. Updated the main reply.
76 karma · joined April 11, 2020
In June of this year, my entire team was struggling after moving to a new Kafka cluster because of low throughout in one of the backend services (about 250k/min with approx 32 instances). This was causing issues for our downstream dependencies as we were not able to reply back within one hour of consuming the record from upstream and our L2 support was daily getting numerous pages which were also escalating to us. Then my manager asked me to take a look at what is going on and if we can improve the throughput somehow. After almost spending one week of banging my head against different theories and tons of experiments in QA I finally figured out that the new Kafka client we were using had a setting where it required acks from all brokers (which has increased to 5 in the new cluster from 3 in previous one) and this caused a huge increase in the blocking time even though we were using an Async framework. The async task just waited too long for completion and once completed could not get the threadpool back due competition with other threads. Solution: simple, I just changed the Kafka producer ack from "ALL" to "ONE" requiring acknowledgement from one broker only. Throughput with same 32 instance jumped from 250k/min to around 700k.
I ask the intelligent readers of HN - do you think that based on this I should be penalized since the change was only in one line of config code? Yes, that's all what it took to resolve this outstanding issue - one line of config change/one commit albeit one week of thinking and experimenting time!
1) First, Scala is designed and developed by "ivory" tower academics primarily for research/PhD thesis purposes. Nothing wrong with that. But that also means language features are added not to solve a practical problem but mostly to show that "it can be done"/"prove a point". When you are awake at 3 AM early in the morning because a release made in the evening went awry and you are trying to grep though logs/code to see what caused the issue I can assure you that debates about "pure" versus "impure", Monads, Kleisli arrow will be further from your mind. So the problem that Scala seems to suffer from is that of "image".
2) Second, Scala is neither a fully functional (FP) nor object oriented (OOP). Support for mixed/multi-paradigm sounds nice on paper but let me ask this question: do you know of any hard core FP programmer who is excited about inheritance or abstract classes or other OOP features? Vice-versa the same thing can be said about many OOP programmers lukewarm response towards many functional features of Scala. I believe that support for mixed/multiple paradigms has hindered large scale adoption of Scala. The language seems to be caught at cross-roads not knowing which way to advance next.
3) Third is lack of niche. True, I absolutely love "Spark" and primarily use it to extract/manipulate data from HDFS/Hive storage but I am gradually gravitating towards "Python" libraries due to their ease of use. What about better Java? Kotlin check. What about sidecar systems used by DevOps? Go check. What about a better designed system language? Rust check. The days of pure general purpose programming languages are truly over and Scala will be in great peril if it fails to expand into newer niches. Without broadening its programmer base it will finally run out of people that can design newer language features in light of ever advancing progresses in processor, OS and memory architectures.
Just my two cents.