stewh_

Microsoft has a problem - poor 3rd party software and drivers make Windows extremely unreliable. Microsoft takes the blame, and Windows looks unreliable. Apple takes a different tack, they simply lock down their APIs and ecosystem to avoid this. Is Microsoft trying to go the Apple route - but maintain some openness? Giving 3rd parties core low-level APIs is ripe for chaos.

I had to install Kaspersky on my main laptop since some VPN software imposed a policy that it installed and up-to-date to connect to a contractor's secured network. It was absolutely terrible. It killed my battery, slowed my machine, killed my TCP stack at one point, interfered all the time, and became generally unbearable. It frustrated me so much, I now do all network operations via a secured VM to avoid the Kaspersky curse on my main work machine.

Hadoop etc is not just about raw scale. Scale is relative to what you are hoping to do with the data - so make the decision based on what you want to do with the data.

Hadoop ecosystem makes exploration easy since it'll support any type of computation (graph, text mining, ML model building/validation). If all you're looking to do is random-access of the data with a small number of known filters/joins at large scale, or you're under no time constraint to explore, then sufficiently optimised DB will be most efficient (and probably cost effective way). Hadoop is a trade-off for general purpose everything, but at cost of infra complexity and computational inefficiency.