I want my physicists to spend their mental effort on physics, not on software architecture.
Physicists are not mathematicians, and yet they are required to acquire a relatively high degree of proficiency in maths because maths is a fundamental tool in their job, and nobody would argue otherwise.
The attitude of considering programming a mundane craft to be picked up as-you-go is the main reason why the scientific software landscape is such a shitshow.
/rant
Just like if you are standing on your two legs most of the day and sprint once in a while, you can be considered a runner. Sure, you can play with semantics, but most people cannot run a marathon.
> The attitude of considering programming a mundane craft to be picked up as-you-go is the main reason why the scientific software landscape is such a shitshow.
Err... That's kinda my point?
> And as such you should be expected to become a decently proficient programmer.
It's very, very hard to be good in 2 different fields. Most people won't have the ability or the context to do so. Even if they did, the time and energy spent to do so would be taken from their main activity, which is why we employ them in the first place.
It's not reasonable to ask a data scientist, geographer, biologist or physicist to follow up with the right practices to deploy the latest sci-stack on a linux server, understand the trade off between GIL locked python thread, asyncio and multiprocessing or spell out what WSGI stands for.
Hell, I know a lot of professional programmers that don't know those things
> Physicists are not mathematicians, and yet they are required to acquire a relatively high degree of proficiency in maths
The quantity of information required to be learned is of one or two orders of magnitude, because the field of maths required to perform physics is quite stable, and well understood.
IT is a very young field, in constant flux. The scientific stack is a moving target, not to even mention the web one. Nobody can expect them to understand python, numpy, pandas, then a web framework, then css, then js, and html, probably some frameworks for them, a builder or two, how to deploy all that stuff in dev, in prod and architectural concerns for linking all that stuff.
That's crazy talk.
If you don't understand the tools you're using, or the environment you're in - you're not any more of a "data scientist" than pretty much everybody else. My carpenter is a data scientist going by this logic.
>The quantity of information required to be learned is of one or two orders of magnitude, because the field of maths required to perform physics is quite stable, and well understood.
Apart from the fact that some areas of physics are really at the forefront of maths, this also ignores the fact that learning the level of proficiency required for graduate work in physics is significantly more involved than learning about some best practices in programming.
I've seen data scientists handling big code bases. The problem was not they couldn't use the language features. The problem is that they would be always lacking essential information for their mission because their is not enough time in a day for a regular human being.
They would put a md5 hashed password in their db, create an xml format to be reusable only to realize they'll need to hard code some value later, or have a gunicorn running to a crawl because they didn't know how to calibrate the number of workers.
It's just too many things to know. Once they mastered that, other things would come to bite them.
It’s definitely a part of it. This isn’t an all or nothing thing, one can learn good practices without encumbering their scientific work.
However, the programming education in science degrees is absolutely appalling. Just show them how program a newton raphson method in matlab (without any considerations for performance) and expect them to know how to program.
It doesn't mean of course that everybody is supposed to be an expert programmer, but a minimum effort to help your colleague is surely not too much to be asked.
Beginners not being aware of some best practices doesn't automatically make them not-programmers.
I work as a data scientist and I see it as part of software development. It's just a different domain - some people do front-end, some do mobile or embedded, I do data science.
Just like I'm not a data scientists, I'm a programmer.
Now, I can use pandas in a pinch and makes pretty graphs, but my statistical analysis will never be on part with yours.
Just like a pianist hobbyist will have a hard time to rival somebody who does that 40 hours a week, although he may be able to play a few fantastic pieces.
Hell, even a web dev programmers, if ask to code a GUI desktop app, is not going to do a good job.
IT is becoming a very large field.
And scientists are not even from this field.
That's not my point.
Again, if it's not possible, then you accept the imperfection of the result, or provide better tooling.
There is no blame to put on them whatsoever.
Similarly, they should hire experienced, qualified software engineers to write/check the software in their papers.
They don't because 'everyone can code - its just logic'.
You might be mixing up the terms. I don't think that the point was about "scientists" in general, it was about "data scientists". The first is a common term used to describe someone who does science in some professional capacity. The second one is a very broad job title within software which very often includes writing code that ends up in production - at some data science roles that might even be your main responsiblity.
A data scientist is not even a somebody trained as a programmer. Their strong suit is data analysis, and it turns out one of the tool to manipulate data today are programming languages so their do it.
But I as a Python trainer, I train data analyst regularly, and they don't have a clue about language ecosystems, how the OS work, data formats or reliable software architecture.
They mainly want to output their graph, pdf report or other media to serve their conclusion. They may want to create some reusable algo, or machine learning model, but that's the limit most of them hit.
If one take their code and put it in prod (which I know happens, don't get me wrong), that's not the data scientist fault. They are doing their job, in which programming is just one of the many means to an end, and is not their specialty.
This is like teaching some JavaScript to complete beginners at a bootcamp and then declaring that front-end developers aren't real programmers because they know so little.
I'm not talking about complete beginners. I don't train beginners.
I'm a physicist (not "data scientist" though I work with plenty of data), and I've been programming since 1981. Anything I do, I want to do well, especially if I do it regularly or it could cause problems if done badly. I've made an effort throughout my career to keep up with good programming practices. I do that out of a combination of pride, curiosity, and professional ethics.
But I'm not a software developer, meaning that I don't create software for widespread or long term use by others. We have an entire department for that, and many of their techniques are quite specialized.
Naturally it wouldn't surprise me if further improving my skills also moves me closer to being capable of software development, and I'm happy to learn and apply their techniques at a pace that works for me. I think that a scientist who is capable of learning to program should receive guidance on how to do it better, but perhaps in stages, such as:
1. Writing code that has a better chance of working, even as it gets bigger and more complex.
2. Working with others on projects that involve sharing code, meaning that it has to be readable and conform to agreed upon standards.
3. Creating code that can be confidently "shipped" for widespread or long term use.
Whether you choose to call it "good" or "not too bad", there is a minimum bar of competence that data scientists need to meet in statistics/ML/AI, programming, and their domain of application. And the ability to move from Jupyter notebooks to Python modules/packages is a basic.