Its something like how baking cookies at home and running a cookie factory are very different. To a person doing each, the behaviors and priorities of the other seem strange and it’s easy to for one to think “we are both just making cookies, why don’t they do what i do which is obviously superior “
Scientific programmers are solving problems first, not writing programs. They are solving problems in a way that is useful only to them or peers who know a whole lot about the problem being solved. The problem-first tools look strange because they deemphasize the programming niceness in favor of problem niceness.
You find the same sort of confusion when programmers are facing business types and excel usage.
There are certainly times when a piece of code starts needing the programming touch, but the right tool for the job depends on the job.
You give good reasons for why a certain situation exists in scientific communities, but I see no reason why it has to be that way.
The various conventions around "accepted" programming paradigms like OO, inheritance, scoping, etc are natural if you use them every day for years, but if you're more interested in optimizing some finances or solving a physics problem, something "primitive" or "messy" like a spreadsheet formula or fortran might actually be more understandable.
I appreciate your point. In most contexts, such as your comparison between Japanese and English, I agree with it. However, paradigms like OO, inheritance, scoping, etc. are hard-won, intellectual accomplishments; they're not arbitrary. They're purposefully designed to solve specific problems. My experience with R showed it to be rife with problems that have been avoidable for decades. AFAICT, it boils down to the tragic view of "I'm a (statistician|engineer|mathematician) not a coder so I don't need to care". The unfortunate truth is that despite such a view, doing analysis with R makes the analyst a programmer by definition. And so the language, ecosystem, and, consequently, users suffer from half-baked, poorly designed workarounds to problems which have long been solved (and abundantly documented in the software literature). To reduce it to a matter of perspective feels to me like a tautology: it's easy once you get it.
Clearly, I'm triggered by this. I hope I've expressed myself respectfully. My point is, I don't feel it's arbitrary. The design and complexity of R has real consequences in terms of cost and reproducibility.
R does have inheritance by the way, not as you will see frequently on "general purpose" programming languages, thing R is not.
[1] http://catern.com/inheritance.html [2] https://books.google.pt/books?id=ka2VUBqHiWkC&pg=PA81&lpg=PA...
"Inheritance" is really the expression of Abstraction in code. Superclass is more general, and thus the abstraction of its subclasses.
Although Smalltalk did not have a specific keyword for "Abstract Class" it used the convention of calling "subclassResponsibility" on methods defined in abstract subperclasses which had to be implemented in subclasses.
Abstraction, generalization, these are conceptual tools. I'm not sure how "composition" models abstraction if at all. Only OOP does.
And one could think that abstraction is a tool valued by scientists. No?
Everything I was taught when OOP became mainstream in the late 80s, was that it was a performance trade off to afford code organization insight, while preserving encapsulation.
This statement about “performance reasons” is quite baffling.
I disagree. There are inherent qualities associated with systems that happen to be easier to understand. If you or I were designing such a system, and wanted to ensure this system imbued similar qualities — I believe it would be possible to do so.
> Japanese is "difficult to understand" if you only know English, but not if you grew up speaking Japanese.
Needing a priori knowledge to be able to understand something, is different.
Can you formalize these inherent qualities? Until you can, it's hard to prove that a quality is or is not essential.
There is an obvious one, though it's tricky to fully formalize and quantify: locality (also known as "coupling"). You ask yourself a question, "if I were to make a small change to a single piece of this program, how much of the program would I have to change/retest/keep in mind?". If the answer is, "typically, just the area around the change", it's a high locality (loosely coupled) program, a good design. If the answer is, "usually, most/all of it", then it's low locality (tight coupling), a bad design.
The reason is, of course, that human mental capacity is limited and in general, number of interactions grows superlinearly with the amount of interacting components. So this here is an objective measure of "easy/hard to understand": the more code you have to keep track of when investigating a random piece of the program, the more mental effort you have to exert - the more difficult to understand it is.
It has some comments about why good software engineering leads to good software, how good software engineering is hard right now because there's a lot of bad examples which cause people to make more bad code, and some suggestions on how things could be done.
Especially consider that much of the audience for R come from proprietary and extremely expensive languages like SAS or SPSS Syntax, neither of which are significantly less difficult to work with but significantly more limited.
Something more intuitive would be nice, but I'm not aware of another option that has the depth of libraries for practical every type of data analysis that can also begin getting a researcher meaningful results in few lines of code than they have fingers on their hands. (Assuming there was no catastrophic accident resulting the loss of fingers.)
Almost all the progression in that industry is relatively straight forward and would make sense to the lay person.
The issue isn't just specialized knowledge - knowing the principles underlying the process, knowing that there are principles - but also access to tooling. I'm sure plenty of lay cooks would appreciate appliances that let them be more consistent, even without knowing any related formal theory - but these are not available. As it is, an oven that's not shit is a major capital investment. It only makes sense to get one if you're more likely to call your work output a "formulated product" than "cookies".
However, software development is a peculiar occupation - one in which the best tools are free. It's like being able to buy a production line for less than a consumer-grade stove. Access to quality tools is not a limiting factor. Knowledge and awareness is.
Of course, non-programmers that code have more interesting things to do than to study software engineering. Which is why it's doubly important to make sure tools that are being popularized aren't shit. On the contrary, tooling is the perfect place to encode good practices and principles, so that even most lay (programming-wise) users succeed by default.
And they don't value writing programs. Or Software Engineers.
That makes productizing some research interesting. That also makes trying to get the same result as something published by your own lab a year ago an uphill battle: "it worked on John's old laptop, the huge Alienware. Never worked on any of our machine".
You think web developers are just writing programs for the hell of it? They have operational constraints around their work as well, it's just that there's a rad open online culture of continuous process improvement that's leveling up the tooling and practices that makes what was once hard look easy.
Things scientists using computers can learn from people working in the software industry:
- use a version control oriented workflow
- write extensive tests
- data management, automated data processing/cleaning pipelines, backups
- build generic frameworks
- use continuous integration
- logging and error reporting for long running tasks
- use modern tooling for automating infrastructure and big jobs, rather than manually submitting Slurm jobs via SSH
#notallscientists but the average level of competence is not good and I think the idea that writing software for scientific research is somehow special is backwards and counterproductive.
I don't see that many job posts doing this anymore. And even then it's just not valued that much.
Not all software development should involve writing your own framework, and it's a bit of a trope that junior SWEs try to write DSLs and frameworks where they should just write some throwaway code. That said, if there's a common modelling or simulation method in your field and there's no great open source generic framework for it, then you should take a hard look at what you are all spending your time on (endlessly reimplementing the wheel).
Is that going to be robust to whatever new cluster comes along five years down the line, or will it all have to be re-written?
I think the reason you and I dislike R is because we just work differently than non-programmers. Non programmers think in purely imperative, straight-forward semantics. They write one-off unmaintainable code tying together libraries that solves their immediate problem. Programmers try and write R code as if it was a proper programming language and immediately run into walls. Non-programmers never see the walls because they don't even know there's another way.
But that's because neither R nor Matlab are primarily programming languages. They're primarily mathematical exploration tools.
Fundamentally, R and Matlab are probably best comparable to Perl in terms of how they got popular. While they are rather ugly, they were the first to provide a simple solution to a few specific problems that a lot of people had, which snowballed them into popularity and network effect advantages
Another common misconception is that AOT compilers can only be used to build libraries/executables, which then get executed. Again, SBCL is an example of an AOT compiler that works at function-level granularity[0], in accordance with Lisp heritage of image-based development, where you write your program by starting it, and then adding/removing/modifying code into a running instance. SBCL achieves this by having the compiler being a part of your application runtime.
I don't know how Julia is implemented internally, but since it's essentially a Lisp clothed in syntax that's more pleasant to the masses, it inherits a lot of Lisp family heritage. That could easily include a fast, to-native, in-process AOT compiler.
--
[0] - Or maybe even lower.
So every function is a multimethod that gets compiled the first time any combination of types is first encountered, and at that point the type information is propagated to all function calls inside the method body, to eliminate dynamic dispatch on those & possibly inline the appropriate method. Julia has parametric polymorphism as well, so it's common for all types in the body to be inferable.
As a result, Julia code tends to make much more heavy use of polymorphism & multimethods than common lisp (where CLOS has a significant runtime cost instead of being a near-zero cost abstraction), since the language and the runtime were designed carefully together to make that fast.
R is statistical-slant, so you have all sorts of distributions and statistical features.
Matlab is general mathematics (but especially matricies). So you can grab say... Galois Fields between 2^2 through 2^16 and just start working with them immediately.
Matlab itself is decently replicated by Octave (which is a great project), but it should be noted that Octave aims to be largely compatible with Matlab (so all of my complaints about how awful that programming language still applies).
There's zero mystery to this. The intended audience is people that want to get stuff done. Professional software developers commenting on R are like this: "He's such a good salesman. He does everything the right way. He dresses right. He talks right. He has the best smile of any salesman I've ever seen. He fills out his reports on time. Granted, he doesn't make many sales, but why would you hire one of those other guys over someone that's perfect?"
If you tried to use R for an actual engineering project you'd wind up with a system that is always on fire. Where as these one off analysis it's fine because it's just a one off.
We want to get stuff done too, but we need it to hold up for more than a few hours so that's the mode we think in.
Programming has a very low barrier to entry and that is incredibly powerful but there is a big gap between someone writing one off scripts to solve immediate problems and people writing highly connected systems that withstand scrutiny.
I've always personally been for accreditation for software engineers much like pretty much every other engineering discipline.
People also say it for k8s. And it kind of explains why k8s is creating so many jobs.
Is it 'creating' jobs? I think it's merely making it easier to specialize.
A few problems are unique because of container usage, but by and large K8s is trying to do what's otherwise a difficult job. Try assembling your own distributed container system with scheduling and whatnot and see if you can build something easier to understand or that works better. Maybe you can, but there's inherent complexity.
The criticism of K8s should really be criticism of indiscriminate container usage and the attempts to ship a company's organization chart as microservices. Many applications should really be monoliths and would work better that way. Some should be split on different "services" (not _micro_) along obvious interface points. Just a minority should be architected as microservices from the get go. Distributed systems are _hard_
For the things it's good at, it's great. For everything else I avoid it like the plague. More often I find it easier to use a quick Python script to generate a data table that I can then read into R to perform whatever stats or plots I need. It's almost always faster than if I had just run everything in R to begin with.
But I think your description is spot-on. Non programmers just want to get something done and if it works in R, then great. For those of us that think in terms of software engineering, good practices can be difficult in R.
The vanilla-R vs Tidyverse split has made this all worse too. These are two completely separate dialects of R that while still the same language, are completely different.
It's like the R folks took the "there's more than one way to do it" mentality from perl and said "challenge accepted!".
And I am saying this as a recent RStudio convert. I love how easy it makes some otherwise hard things. I hate the sheer amount of hoops you have to jump through to make some stuff work.
But it works and I am definitely not an engineer.
R has had a very long evolution. It is a very different beast today in its most common usage, than it was the earlier days 10, 20+ years ago. Even the Tidyverse has some libraries that are very much crafted with a programmer mindset like purrr and tidyr. These tools are decidedly non-imperative and not straight-forward in their semantics.
What makes R difficult for experienced programmers, I think, is the inconsistency of paradigms that are the result of its long history. This complicates how one writes library code.
There is, however, a "sweet-spot" for R and that would be as a "notebook" based programming language much like Mathematica, Matlab, and Julia. Which one you like, I guess, depends on your taste, your own history, and the killer libraries you want to use.
Whenever I have to describe what R is all about to excel jockeys at work, I just say it's "excel on sterioids". I think that's fair (albeit reductive) description. To be honest, I probably would have never learned R if Julia had existed when I started picking up R. I think I would have preferred a more ahistorical language with less "baggage" than R. But it's always worked out for me, so I am sticking to it at least for now.
Maybe you'd like this post: http://bioinfomofo.blogspot.com/2014/01/r-hard-things-are-ea...
Now that I’m used to both languages, I find it funny how much R is hated by “true” programmers.
R only had to be ergonomically better than the competition, and they weren't very good.
Off-by-one dimensionality errors are so common in programming. If the language does something like zero-extending instead of raising an error, it will lead to an "it runs but gives the wrong answer" bug. These are much more painful in numerical code than in logic-based code.
In fact, I would probably expect the output to be (1,0,1,2,3,4,5,6).
For instance, for the statement (a,b,c,d,e) + (a,b,c,d,e,f) are we really all saying that it’s clear, obvious and unambiguous that this means (a+a, b+b, c+c, d+d, e+e, f+a).
Personally if you showed me that and asked me to describe what would happen before reading this thread, I would have said it would either throw an error, concatenate the vectors or only add the matching elements. I wouldn’t personally guess that the first vector would be treated as a repeating sequence, but different people might make different assumptions - I just don’t think it’s particularly clear, and I struggle to believe it’s clear, unambiguous and obvious to statisticians too.
Heh this itself is common evidence of an (experienced) programmer's way of thinking. Remember, when dealing with mathematics, there's no machine (or runtime or similar abstraction) there lurking in the background, enforcing conditions, or even lending a physical reality. Operations in mathematics are defined. Notation in mathematics is just an operation encoded as an "arbitrary" symbol. Which leads to
> I just don’t think it’s particularly clear, and I struggle to believe it’s clear, unambiguous and obvious to statisticians too.
This happens to be convention in both a subset of textbooks and many programming environments. It's mostly an artifact of the notation.
If you want to explore what math notation "feels like" a bit more without learning math (which I _wholeheartedly_ recommend as it's incredibly useful), try out the APL programming language a bit. It evokes a similar atmosphere of notation conveying the idea of well-defined operations.
The result should probably be vector promotion then (2,2,3,4,5,6). (2,2,4,4,6,6) is not a good answer.
That is the plus symbol and it is in the context of two vectors. The intuitive thing to do is to fail with an error if it is a mathematics operation, concatenate if it is a programming context or treat the two vectors as the same length by appending 0s to the shorter one if the goal is to be unhelpfully helpful.
Repeating a vector until the dimensions match then element-wise adding them may be convenient for you, and you may like it. Maybe even lots of people like it. But it is a tough sell for me to believe it is intuitive.
numpy calls it 'broadcasting', although this particular operation doesn't work. For example, this is valid in numpy:
`np.array([1,2]) > np.array([[1],[2]])`
The idea is that an attempt is made to handle the operation even if the arrays / matrices are incompatible. Extending the operations of matrices and vectors in this way allows for extremely concise operations that would otherwise be a pain, like in the original R example.
> c(1, 0) + c(1, 2, 3, 4, 5, 6)
[1] 2 2 4 4 6 6
>>> numpy.array([1,0]) + numpy.array([1, 2, 3, 4, 5, 6])
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ValueError: operands could not be broadcast together with shapes (2,) (6,)
np.array([1,2]) + np.array([[1],[2]])
is not broadly defined and understood in pure mathematics. Looking at it, it's not immediately obvious what it would do. You are adding a (1,2) matrix to a (2,1) matrix. It turns out, numpy extends the rules of matrix multiplication to matrix addition through the rule of broadcasting.
R is just doing the same thing, in a different way. These types of implementations are common in scientific computing because it creates easy ways to do these operations, programming robustness and rigidity be damned.
Also, the cheat sheets put out by the RStudio team are the best programming language cheat sheets I've seen for any language: https://www.rstudio.com/resources/cheatsheets/
I don't do R much anymore (at the end of the day I personally have the freedom to start fresh so I choose that over dealing with technical debt in the R language itself), but the R Studio product, team, and R community I found fantastic.
I'd say the learning curve for making a sexy plot is a lot shorter with tidyverse, but overall, relying on it handicaps you versus spending the half hour longer to do the same thing with base graphics (or a base-like package).
Well, yes, but that is part of the reason why it is so popular. R may be written by world-leading language design experts - but it doesn't show!
> ...relying on it handicaps you versus spending...
There is almost no reason to drop out of tidyverse if the problem domain is visualising data. Most people could go an entire career as an analyst using just ggplot + tidyverse.
If something needs to be plotted and ggplot won't work it probably makes sense to drop out of R and go straight to TikZ or OpenGL.
or D3.js
You are either going to use a programming language (or library, etc.) made by a programmer pretending they know about statistics, or a statistician pretending they know about programming. Oftentimes, as a programmer, the right choice is the former, but not uncommonly (because statistics is even less intuitive than programming), you really really need to know that the statistics have been done right. If someone has ported the relevant code from R to python, great. If not, bit the bullet and use R, it's where the statisticians hang out.
You know, I bet statisticians don't think any more kindly of how programmers make stuff. Our use of the '=' sign, for example. We're just used to that kind of thing, so it doesn't look like a problem to us.
That said, I feel like correctness should be given a higher priority in scientific computing and yet a dynamically typed, lazily evaluated language is used.
R is a terrible programming language.
It's not a bad language for invoking code, because for many of those people, they're not taught the concepts behind any language, so it's all semi-arbitrary symbols.
model <- lm(outcome ~ variable1 + variable2 + variable3, data=data)
summary(model)
Isn't any more complex than anything else. And what R does have is a network effect - at this point, for almost any statistical task I've ever encountered, there's R code for it.
Sure the syntax is going to seem alien to them, but so would any first encounter with a programming language.
The use case for R simply isn't a traditional programmer. That isn't the target user. Sure if you need an application that might need significant scale you're not going to use R Shiny, but a lot of R work are one-off bespoke analysis projects. Models that do need to be deployed for use at scale in an application take their output parameters from R models and simply implement them in the app. I do this myself, taking coefficients etc, implement a function call in a database and then use the results on the front end.
The object system mess though... I have no words.
software engineering practices don't exist unless you have a gigantic program, programming language theory is either not interesting or too foreign for them
about how they manage.. it's easy, they get used to it
Now, most people just use Python.
R is very popular with stats people but new PhD candidates are beginning to write python implementations of R things (sort of how like DataFrames/pandas happened).
People have been saying this since I was an undergraduate.
I'm submitting my tenure packet this year.
> I'm submitting my tenure packet this year.
That doesn't mean it isn't happening. If anything, the fact that the rate of adoption is slow over a long period of time would tell me that there's more stating power here. If suddenly 20% used python that would be a warning sign. If there was .5% year over year for decades.... That's different.
and also "The year R unseats Python."
https://www.datanami.com/2019/08/15/is-python-strangling-r-t...
https://madnight.github.io/githut/#/pull_requests/2014/1
R has halved in percentage since 2018