The software tools: For me, PyCharm and Jupyter for development, Dask for data preprocessing, Tensorflow, Theano, and various scipy libraries for ML, as well as helpers such as PyMC3 and Edward for probabilistic programming.
The important tool: a PhD in applied mathematics that taught me how to think originally and creatively about problems, to abstract and codify them with mathematics, and to approach them numerically and computationally.
I suspect that the answer you were looking for is the first set of tools. This set is extremely nice to have; the second set, at least for me, is essential.
I find it a bit disheartening that this is how you read my response to OP's question. The denizens of HN may in general disapprove, but I believe my education to be one of the most important factors in doing my job. I don't think it's fair to respond with thinly-veiled accusations of elitism, especially when I explicitly tried to also give him or her a "practical" answer.
> Nevertheless, 80/20 (80% solution in 20% time) does wonders with the right set of tools and a rudimentary appreciation of applied maths, 80/20 being often what businesses really need.
In some cases, you're right, but more and more often I see people assuming that they've found the "80% solution", when in reality it's something like a 30 or 40% solution. My personal experience is that with many problems, a bit of thought shows you that many seemingly simple problems are hiding both a lot of complexity and a lot of potential - and I've seen real money left on the table because of a failure to recognise this.
My opinion is that conceptualising the problem in the first place is what allows you to see really what an "80/20" solution looks like, and in my case I don't think I would be able to do that without my most potent weapon: my mathematical education.
Not every problem is like that: sometimes a simple, out-of-the-box solution does work really well, and solves a concrete business need. My point is that when this is about the limit of what you can do, you don't really know whether your problem is in this class or not.
When you say "PhD in applied mathematics taught me how to abstract and codify with math, and to approach numerically and computationally", do you feel you already had good mathematical intuition earlier too and the PhD made it better? Or did the PhD by itself help you build such skills from scratch?
I too would like to have such skills one day, but I don't have the opportunities or time for a PhD. Do you feel it can be self taught to some extent?
I really think that the PhD experience was the major factor in me developing this, though I think I showed a certain promise earlier on. I think it's certainly true that, without the PhD, I wouldn't have known how to go about the process of learning and thinking in the way that I do now.
> Do you feel it can be self taught to some extent?
I think you'd have to be a much smarter and more dedicated person than I am to achieve the same results without the environment, and as I mentioned in a cousin comment, I think that environment might be tough to replicate. I am a big proponent of formal education for exactly this reason. Academia certainly has many flaws, and I'm a harsh critic of them, but I personally experienced what I consider to be the best side of it, and it's this side that formed the opinion that I'm expressing here.
EDIT0: Repeating what I posted in a cousin comment: Also worth noting that by far the smartest person I know - a deep thinker, also nominally a data scientist, but foremost a truly original and creative mathematical thinker - didn't complete his PhD and left with a Masters. Intellectually, this guy makes me look like a geriatric Corgi on opium, and he didn't need a PhD to make him that way.
What is a first approximation to this experience, particularly the creative thinking portion? As someone with a MS but without a PhD, I suspect you get to flex this ability during the research phase of your studies.
* Spending several years, uninterrupted and undistracted, thinking deeply about a mathematical topic.
* Being advised and supported, but not led or coddled, by an experienced researcher.
* Being surrounded by clever people, who do research full-time.
* Working in an environment where not knowing the answer. immediately, or even quickly, is not just accepted by expected.
* Being "left in the dark", with the understanding that the process of being in that position and finding your way out is the very thing that you are trying to learn.
* Having rigorous expectations for the "final result" (to the extent that one can be found).
I can't think of a great approximation to that, but perhaps someone else can chime in.
EDIT0: Formatting
EDIT1: Also worth noting that by far the smartest person I know - a deep thinker, also nominally a data scientist, but foremost a truly original and creative mathematical thinker - didn't complete his PhD and left with a Masters. Intellectually, this guy makes me look like a geriatric Corgi on opium, and he didn't need a PhD to make him that way.
Daily I'd say RStudio, Excel, Tableau, PowerPoint. Either I'm coding in R or I'm presenting.
Otherwise jupyter with numpy and all the other great python libraries is where I spend much of my time.
That said: I think often these 45 minute tasks should be much faster, and folks would actually benefit from optimising their code a bit more. If I have to do a word count in 30gb of text, I'd rather implement carefully or use Hyper Log Log. I find it weird that people would rather use Spark, which I always find quite painful. I guess it comes down to familiarity.