Complexity is a source of income in open source ecosystems (2019)
r-bloggers.com
r-bloggers.com
> Matloff is also upset about a commercial company swooping in to steal their precious, a common academic complaint (academics swooping in to steal ideas from commercially developed software is, of course, perfectly respectable).
This reads like some kind of sarcastic statement. In fact it is indeed perfectly respectable, that academics do this, because it lets our society advance and research progress outside of proprietary software gardens. In my opinion any such kind of action is perfectly respectable, if it shares knowledge with the world. The other way around it is not OK, because from proprietarizing ideas, there is no benefit for society. If a company is worth its salt, it will offer services on top of existing public knowledge, without creating walled gardens. If it cannot do so, then it might be time to realize, that the company should not exist.
What is not OK is, if academics take that knowledge and make it seem like they were the ones inventing stuff. One should always give credit where credit is due. Never forget academic honesty. Usually however, a lot of foundational research is done by academics, only to be picked up years later and often by commercial entities dealing in proprietary software.
I contend that every profitable company leveraging public knowledge does offer services (broadly construed) on top of that knowledge, since otherwise no one would pay them. Maybe the minimal example is that in which the only service provided is marketing, but the range of service levels provided in practice runs the gamut. RStudio clearly goes well beyond that minimum.
In some cases, the services offered are so valuable that the company can create a walled garden - which reduces the net value of those services to the customer but provides the company with monopolistic pricing power. (It’s not clear to me that RStudio’s products have significant lock-in, for what it’s worth. R has significant lock-in, like any programming language, and RStudio benefits from the lack of competitors creating good R tooling, but that’s not really a walled garden of RStudio’s making.)
Personally I think RStudio is far less clunky than, say Jupyter. I still think Matlab is by far the best tool for engineering. I think having a company supporting the technology like that has generally been very beneficial. It's certainly possible they would just cause it to stagnate and milk it for cash as the technology dies out, but their business would die with it. Hence these examples have never stopped advancing.
Sure, but people can have different views about where that point should be. Documenting industry-relevant know-how makes it even easier for folks to take the result and run with it (providing the 'impact' that most funders will ultimately care about), so I agree with GP that it's very much a proper field for academic research.
A company commercializing an idea from academia does not make that idea proprietary.
It's an important point, but only about abuse of the word "steal".
> A company commercializing an idea from academia does not make that idea proprietary.
It doesn't, up until they start armoring it with their own IP until nobody else can reasonably extend the original idea anymore...
You say this as if proprietary rules (ie. copyright and patents) weren't intentionally designed to incentivize knowledge creation & sharing by ensuring compensation for innovation. Creators can't pay bills with societal advancements and research progress. This is a really shallow "proprietary = bad" ideological take that I expect to see on the black & white pages of Stallman's website, not an (at least partly) entrepreneurial forum.
tired: IP = bad
wired: IP rules are "intentionally designed to incentivize knowledge creation & sharing by ensuring compensation for innovation"
inspired: above isn't true in practice; IP rules have been thoroughly gamed, and their primary job is protecting monopolistic behavior, even if it means keeping knowledge suppressed past the point it becomes irrelevant or useless
Copyright and patents, as practiced today, have mutated into a malignant form. That doesn't mean the idea is bad, just that it was taken too far - instead of creating idea-generating pump, we've ended up with toolkits for extracting rent from society and creating wasteful artificial scarcities.
Whatever technological / knowledge advancement any entity holds proprietary could probably be of more use when being shared with the world. Our technology enables us to have this kind of knowledge society, where knowledge is accessible at all times almost anywhere. I don't think that stopping knowledge from spreading can be a net gain, just because it enables a few selected individuals at some entity to gain from keeping it a secret. Such a thing should not be the basis of a business.
Apparently all you have to say is "stopping knowledge from spreading = bad", yet in the same breath you chastise businesses for taking from academia without returning any form of compensation. I'm not in academia and I'm not sure if you are, but best of luck to them. Academia is shrinking/slowing as they beg for government scraps and flee to proprietary business.
The idea is, that it is very unlikely, that there can be a situation, in which society as a whole gains more from only one actor having the knowledge, than the knowledge being available for everyone (and subsequently more people being able to work on progress using that knowledge). If only one actor has the knowledge, this will most likely delay progress, because this actor does not need to innovate. They can sit on their behind and profit from that knowledge, which they do not share. In comparison, if the knowledge was shared, then it would be about being the first to get to the next level, in order to be the first, who can adapt to progress. The first one to offer a new service based on that knowledge or some technology resulting from it. If you are the one making the progress, you will have a head start on everyone else.
(If you want an up-to-date example: Vaccines. If big pharma businesses had to share all knowledge during the whole research process, it might be, that we could have a better vaccine and could have had it earlier than we actually did.)
So yes, I see stopping knowledge from spreading as a behavior, that has a net negative impact, when compared to a society, where we do not stop knowledge from spreading. Most certainly a society that stops knowledge from spreading will fall behind.
Then another point is, that I am not chastizing businesses for taking from academia without returning any form of compensation, as you state. I am chastizing them for taking and then making some profit of of unshared knowledge, that they gain. It is not about paying money or compensating in any way academia, but about sharing their gained knowledge with society, with the public.
I am thinking of a quite different business world here, where business actually and exclusively serve the purpose of advancing society, as so many of them claim. This is brought about by having a strict knowledge sharing policy and providing actual services, instead of holding people captive, with their proprietary tech and secret knowledge. If a competitor surpasses them, great! Someone found a better way! Society advanced! That is the kind of mindset I am thinking of.
Businesses in my opinion should not exist to make individuals rich based on profits, which are based on keeping knowledge away from others, who could use it well, even if for a competing product. If a business makes anyone a rich person, then it should be based on that business providing beneficial services for society.
It could happen, that a business is created merely for providing a narrow service on top of some tech, and never makes any progress. It never gains any new insight or knowledge. Perhaps, because this business does not have a research and development department. That is totally fine! As long as people find that service useful, such a business can exist. It might not grow infinitely. Might even remain a small specialized business. Perhaps this business has to evolve the service at some point, but it can still be on top of existing public knowledge.
For example a business could be to set up IT infrastructure for private or other businesses. That is a lot of work, which not everyone wants to do themselves. And then to make it reliable for the specific use case ... Yet there is no new research being made by that company. It is the good quality of their work and happy customers, that generate their income. They never return anything to academia and I am not criticizing them for it.
I do not want to stick to what exists today and how things work today. I am imagining how a better situation could look like and how it could work. This is about discussing ideas and not about how any particular business, that exists today, could be saved, if the business world was restructured in this way from one day to the next. It is quite likely, that many businesses would fail, without keeping knowledge a secret. I say good riddance!
Except for a profit motive for companies which then benefits society by lowering prices of products. If it weren't for private companies no one of us could afford a computer today.
Maintaining a market that's beneficial to society is an attempt to carefully tune interplay between many public and private concerns. From the point of view of society, the market is just a roundabout way of putting people on a treadmill and hanging carrots in front of them. We need companies to produce goods, provide services, and innovate better solutions, so we allow a degree of privileges and exclusivity for things that otherwise shouldn't be owned - like ideas. But we cannot allow companies to actually get the carrot and stop running - because it's the production and innovation treadmill that's the important thing (from society's POV).
This model has been so successful that it blinded a generation or two of politicians - they let the donkeys get their carrots, and now we have fat donkeys saying to us, "if you want us to run the treadmill some more, give us the carrots".
Might need a (2019) tag. As of 2020, RStudio is a Public Benefit Corporation and has a corporate structure that is designed for long-term investment in the scientific community, and isn't susceptible to buyout or IPO.
https://blog.rstudio.com/2020/01/29/rstudio-pbc/
(disclaimer: I work at RStudio)
Is there any information you can share on what steps RStudio has taken to reduce susceptibility to buyout and IPO pressure from VC?
I think the reason you and I dislike R is because we just work differently than non-programmers. Non programmers think in purely imperative, straight-forward semantics. They write one-off unmaintainable code tying together libraries that solves their immediate problem. Programmers try and write R code as if it was a proper programming language and immediately run into walls. Non-programmers never see the walls because they don't even know there's another way.
But that's because neither R nor Matlab are primarily programming languages. They're primarily mathematical exploration tools.
Fundamentally, R and Matlab are probably best comparable to Perl in terms of how they got popular. While they are rather ugly, they were the first to provide a simple solution to a few specific problems that a lot of people had, which snowballed them into popularity and network effect advantages
Another common misconception is that AOT compilers can only be used to build libraries/executables, which then get executed. Again, SBCL is an example of an AOT compiler that works at function-level granularity[0], in accordance with Lisp heritage of image-based development, where you write your program by starting it, and then adding/removing/modifying code into a running instance. SBCL achieves this by having the compiler being a part of your application runtime.
I don't know how Julia is implemented internally, but since it's essentially a Lisp clothed in syntax that's more pleasant to the masses, it inherits a lot of Lisp family heritage. That could easily include a fast, to-native, in-process AOT compiler.
--
[0] - Or maybe even lower.
So every function is a multimethod that gets compiled the first time any combination of types is first encountered, and at that point the type information is propagated to all function calls inside the method body, to eliminate dynamic dispatch on those & possibly inline the appropriate method. Julia has parametric polymorphism as well, so it's common for all types in the body to be inferable.
As a result, Julia code tends to make much more heavy use of polymorphism & multimethods than common lisp (where CLOS has a significant runtime cost instead of being a near-zero cost abstraction), since the language and the runtime were designed carefully together to make that fast.
R is statistical-slant, so you have all sorts of distributions and statistical features.
Matlab is general mathematics (but especially matricies). So you can grab say... Galois Fields between 2^2 through 2^16 and just start working with them immediately.
Matlab itself is decently replicated by Octave (which is a great project), but it should be noted that Octave aims to be largely compatible with Matlab (so all of my complaints about how awful that programming language still applies).
There's zero mystery to this. The intended audience is people that want to get stuff done. Professional software developers commenting on R are like this: "He's such a good salesman. He does everything the right way. He dresses right. He talks right. He has the best smile of any salesman I've ever seen. He fills out his reports on time. Granted, he doesn't make many sales, but why would you hire one of those other guys over someone that's perfect?"
If you tried to use R for an actual engineering project you'd wind up with a system that is always on fire. Where as these one off analysis it's fine because it's just a one off.
We want to get stuff done too, but we need it to hold up for more than a few hours so that's the mode we think in.
Programming has a very low barrier to entry and that is incredibly powerful but there is a big gap between someone writing one off scripts to solve immediate problems and people writing highly connected systems that withstand scrutiny.
I've always personally been for accreditation for software engineers much like pretty much every other engineering discipline.
People also say it for k8s. And it kind of explains why k8s is creating so many jobs.
Is it 'creating' jobs? I think it's merely making it easier to specialize.
A few problems are unique because of container usage, but by and large K8s is trying to do what's otherwise a difficult job. Try assembling your own distributed container system with scheduling and whatnot and see if you can build something easier to understand or that works better. Maybe you can, but there's inherent complexity.
The criticism of K8s should really be criticism of indiscriminate container usage and the attempts to ship a company's organization chart as microservices. Many applications should really be monoliths and would work better that way. Some should be split on different "services" (not _micro_) along obvious interface points. Just a minority should be architected as microservices from the get go. Distributed systems are _hard_
For the things it's good at, it's great. For everything else I avoid it like the plague. More often I find it easier to use a quick Python script to generate a data table that I can then read into R to perform whatever stats or plots I need. It's almost always faster than if I had just run everything in R to begin with.
But I think your description is spot-on. Non programmers just want to get something done and if it works in R, then great. For those of us that think in terms of software engineering, good practices can be difficult in R.
The vanilla-R vs Tidyverse split has made this all worse too. These are two completely separate dialects of R that while still the same language, are completely different.
It's like the R folks took the "there's more than one way to do it" mentality from perl and said "challenge accepted!".
And I am saying this as a recent RStudio convert. I love how easy it makes some otherwise hard things. I hate the sheer amount of hoops you have to jump through to make some stuff work.
But it works and I am definitely not an engineer.
R has had a very long evolution. It is a very different beast today in its most common usage, than it was the earlier days 10, 20+ years ago. Even the Tidyverse has some libraries that are very much crafted with a programmer mindset like purrr and tidyr. These tools are decidedly non-imperative and not straight-forward in their semantics.
What makes R difficult for experienced programmers, I think, is the inconsistency of paradigms that are the result of its long history. This complicates how one writes library code.
There is, however, a "sweet-spot" for R and that would be as a "notebook" based programming language much like Mathematica, Matlab, and Julia. Which one you like, I guess, depends on your taste, your own history, and the killer libraries you want to use.
Whenever I have to describe what R is all about to excel jockeys at work, I just say it's "excel on sterioids". I think that's fair (albeit reductive) description. To be honest, I probably would have never learned R if Julia had existed when I started picking up R. I think I would have preferred a more ahistorical language with less "baggage" than R. But it's always worked out for me, so I am sticking to it at least for now.
Maybe you'd like this post: http://bioinfomofo.blogspot.com/2014/01/r-hard-things-are-ea...
Now that I’m used to both languages, I find it funny how much R is hated by “true” programmers.
R only had to be ergonomically better than the competition, and they weren't very good.
Off-by-one dimensionality errors are so common in programming. If the language does something like zero-extending instead of raising an error, it will lead to an "it runs but gives the wrong answer" bug. These are much more painful in numerical code than in logic-based code.
In fact, I would probably expect the output to be (1,0,1,2,3,4,5,6).
For instance, for the statement (a,b,c,d,e) + (a,b,c,d,e,f) are we really all saying that it’s clear, obvious and unambiguous that this means (a+a, b+b, c+c, d+d, e+e, f+a).
Personally if you showed me that and asked me to describe what would happen before reading this thread, I would have said it would either throw an error, concatenate the vectors or only add the matching elements. I wouldn’t personally guess that the first vector would be treated as a repeating sequence, but different people might make different assumptions - I just don’t think it’s particularly clear, and I struggle to believe it’s clear, unambiguous and obvious to statisticians too.
Heh this itself is common evidence of an (experienced) programmer's way of thinking. Remember, when dealing with mathematics, there's no machine (or runtime or similar abstraction) there lurking in the background, enforcing conditions, or even lending a physical reality. Operations in mathematics are defined. Notation in mathematics is just an operation encoded as an "arbitrary" symbol. Which leads to
> I just don’t think it’s particularly clear, and I struggle to believe it’s clear, unambiguous and obvious to statisticians too.
This happens to be convention in both a subset of textbooks and many programming environments. It's mostly an artifact of the notation.
If you want to explore what math notation "feels like" a bit more without learning math (which I _wholeheartedly_ recommend as it's incredibly useful), try out the APL programming language a bit. It evokes a similar atmosphere of notation conveying the idea of well-defined operations.
The result should probably be vector promotion then (2,2,3,4,5,6). (2,2,4,4,6,6) is not a good answer.
That is the plus symbol and it is in the context of two vectors. The intuitive thing to do is to fail with an error if it is a mathematics operation, concatenate if it is a programming context or treat the two vectors as the same length by appending 0s to the shorter one if the goal is to be unhelpfully helpful.
Repeating a vector until the dimensions match then element-wise adding them may be convenient for you, and you may like it. Maybe even lots of people like it. But it is a tough sell for me to believe it is intuitive.
numpy calls it 'broadcasting', although this particular operation doesn't work. For example, this is valid in numpy:
`np.array([1,2]) > np.array([[1],[2]])`
The idea is that an attempt is made to handle the operation even if the arrays / matrices are incompatible. Extending the operations of matrices and vectors in this way allows for extremely concise operations that would otherwise be a pain, like in the original R example.
> c(1, 0) + c(1, 2, 3, 4, 5, 6)
[1] 2 2 4 4 6 6
>>> numpy.array([1,0]) + numpy.array([1, 2, 3, 4, 5, 6])
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ValueError: operands could not be broadcast together with shapes (2,) (6,)
np.array([1,2]) + np.array([[1],[2]])
is not broadly defined and understood in pure mathematics. Looking at it, it's not immediately obvious what it would do. You are adding a (1,2) matrix to a (2,1) matrix. It turns out, numpy extends the rules of matrix multiplication to matrix addition through the rule of broadcasting.
R is just doing the same thing, in a different way. These types of implementations are common in scientific computing because it creates easy ways to do these operations, programming robustness and rigidity be damned.
Its something like how baking cookies at home and running a cookie factory are very different. To a person doing each, the behaviors and priorities of the other seem strange and it’s easy to for one to think “we are both just making cookies, why don’t they do what i do which is obviously superior “
Scientific programmers are solving problems first, not writing programs. They are solving problems in a way that is useful only to them or peers who know a whole lot about the problem being solved. The problem-first tools look strange because they deemphasize the programming niceness in favor of problem niceness.
You find the same sort of confusion when programmers are facing business types and excel usage.
There are certainly times when a piece of code starts needing the programming touch, but the right tool for the job depends on the job.
You give good reasons for why a certain situation exists in scientific communities, but I see no reason why it has to be that way.
The various conventions around "accepted" programming paradigms like OO, inheritance, scoping, etc are natural if you use them every day for years, but if you're more interested in optimizing some finances or solving a physics problem, something "primitive" or "messy" like a spreadsheet formula or fortran might actually be more understandable.
I appreciate your point. In most contexts, such as your comparison between Japanese and English, I agree with it. However, paradigms like OO, inheritance, scoping, etc. are hard-won, intellectual accomplishments; they're not arbitrary. They're purposefully designed to solve specific problems. My experience with R showed it to be rife with problems that have been avoidable for decades. AFAICT, it boils down to the tragic view of "I'm a (statistician|engineer|mathematician) not a coder so I don't need to care". The unfortunate truth is that despite such a view, doing analysis with R makes the analyst a programmer by definition. And so the language, ecosystem, and, consequently, users suffer from half-baked, poorly designed workarounds to problems which have long been solved (and abundantly documented in the software literature). To reduce it to a matter of perspective feels to me like a tautology: it's easy once you get it.
Clearly, I'm triggered by this. I hope I've expressed myself respectfully. My point is, I don't feel it's arbitrary. The design and complexity of R has real consequences in terms of cost and reproducibility.
R does have inheritance by the way, not as you will see frequently on "general purpose" programming languages, thing R is not.
[1] http://catern.com/inheritance.html [2] https://books.google.pt/books?id=ka2VUBqHiWkC&pg=PA81&lpg=PA...
"Inheritance" is really the expression of Abstraction in code. Superclass is more general, and thus the abstraction of its subclasses.
Although Smalltalk did not have a specific keyword for "Abstract Class" it used the convention of calling "subclassResponsibility" on methods defined in abstract subperclasses which had to be implemented in subclasses.
Abstraction, generalization, these are conceptual tools. I'm not sure how "composition" models abstraction if at all. Only OOP does.
And one could think that abstraction is a tool valued by scientists. No?
Everything I was taught when OOP became mainstream in the late 80s, was that it was a performance trade off to afford code organization insight, while preserving encapsulation.
This statement about “performance reasons” is quite baffling.
I disagree. There are inherent qualities associated with systems that happen to be easier to understand. If you or I were designing such a system, and wanted to ensure this system imbued similar qualities — I believe it would be possible to do so.
> Japanese is "difficult to understand" if you only know English, but not if you grew up speaking Japanese.
Needing a priori knowledge to be able to understand something, is different.
Can you formalize these inherent qualities? Until you can, it's hard to prove that a quality is or is not essential.
There is an obvious one, though it's tricky to fully formalize and quantify: locality (also known as "coupling"). You ask yourself a question, "if I were to make a small change to a single piece of this program, how much of the program would I have to change/retest/keep in mind?". If the answer is, "typically, just the area around the change", it's a high locality (loosely coupled) program, a good design. If the answer is, "usually, most/all of it", then it's low locality (tight coupling), a bad design.
The reason is, of course, that human mental capacity is limited and in general, number of interactions grows superlinearly with the amount of interacting components. So this here is an objective measure of "easy/hard to understand": the more code you have to keep track of when investigating a random piece of the program, the more mental effort you have to exert - the more difficult to understand it is.
It has some comments about why good software engineering leads to good software, how good software engineering is hard right now because there's a lot of bad examples which cause people to make more bad code, and some suggestions on how things could be done.
Especially consider that much of the audience for R come from proprietary and extremely expensive languages like SAS or SPSS Syntax, neither of which are significantly less difficult to work with but significantly more limited.
Something more intuitive would be nice, but I'm not aware of another option that has the depth of libraries for practical every type of data analysis that can also begin getting a researcher meaningful results in few lines of code than they have fingers on their hands. (Assuming there was no catastrophic accident resulting the loss of fingers.)
Almost all the progression in that industry is relatively straight forward and would make sense to the lay person.
The issue isn't just specialized knowledge - knowing the principles underlying the process, knowing that there are principles - but also access to tooling. I'm sure plenty of lay cooks would appreciate appliances that let them be more consistent, even without knowing any related formal theory - but these are not available. As it is, an oven that's not shit is a major capital investment. It only makes sense to get one if you're more likely to call your work output a "formulated product" than "cookies".
However, software development is a peculiar occupation - one in which the best tools are free. It's like being able to buy a production line for less than a consumer-grade stove. Access to quality tools is not a limiting factor. Knowledge and awareness is.
Of course, non-programmers that code have more interesting things to do than to study software engineering. Which is why it's doubly important to make sure tools that are being popularized aren't shit. On the contrary, tooling is the perfect place to encode good practices and principles, so that even most lay (programming-wise) users succeed by default.
And they don't value writing programs. Or Software Engineers.
That makes productizing some research interesting. That also makes trying to get the same result as something published by your own lab a year ago an uphill battle: "it worked on John's old laptop, the huge Alienware. Never worked on any of our machine".
You think web developers are just writing programs for the hell of it? They have operational constraints around their work as well, it's just that there's a rad open online culture of continuous process improvement that's leveling up the tooling and practices that makes what was once hard look easy.
Things scientists using computers can learn from people working in the software industry:
- use a version control oriented workflow
- write extensive tests
- data management, automated data processing/cleaning pipelines, backups
- build generic frameworks
- use continuous integration
- logging and error reporting for long running tasks
- use modern tooling for automating infrastructure and big jobs, rather than manually submitting Slurm jobs via SSH
#notallscientists but the average level of competence is not good and I think the idea that writing software for scientific research is somehow special is backwards and counterproductive.
I don't see that many job posts doing this anymore. And even then it's just not valued that much.
Not all software development should involve writing your own framework, and it's a bit of a trope that junior SWEs try to write DSLs and frameworks where they should just write some throwaway code. That said, if there's a common modelling or simulation method in your field and there's no great open source generic framework for it, then you should take a hard look at what you are all spending your time on (endlessly reimplementing the wheel).
Is that going to be robust to whatever new cluster comes along five years down the line, or will it all have to be re-written?
That said, I feel like correctness should be given a higher priority in scientific computing and yet a dynamically typed, lazily evaluated language is used.
software engineering practices don't exist unless you have a gigantic program, programming language theory is either not interesting or too foreign for them
about how they manage.. it's easy, they get used to it
I'd say the learning curve for making a sexy plot is a lot shorter with tidyverse, but overall, relying on it handicaps you versus spending the half hour longer to do the same thing with base graphics (or a base-like package).
Well, yes, but that is part of the reason why it is so popular. R may be written by world-leading language design experts - but it doesn't show!
> ...relying on it handicaps you versus spending...
There is almost no reason to drop out of tidyverse if the problem domain is visualising data. Most people could go an entire career as an analyst using just ggplot + tidyverse.
If something needs to be plotted and ggplot won't work it probably makes sense to drop out of R and go straight to TikZ or OpenGL.
or D3.js
The object system mess though... I have no words.
Also, the cheat sheets put out by the RStudio team are the best programming language cheat sheets I've seen for any language: https://www.rstudio.com/resources/cheatsheets/
I don't do R much anymore (at the end of the day I personally have the freedom to start fresh so I choose that over dealing with technical debt in the R language itself), but the R Studio product, team, and R community I found fantastic.
You are either going to use a programming language (or library, etc.) made by a programmer pretending they know about statistics, or a statistician pretending they know about programming. Oftentimes, as a programmer, the right choice is the former, but not uncommonly (because statistics is even less intuitive than programming), you really really need to know that the statistics have been done right. If someone has ported the relevant code from R to python, great. If not, bit the bullet and use R, it's where the statisticians hang out.
You know, I bet statisticians don't think any more kindly of how programmers make stuff. Our use of the '=' sign, for example. We're just used to that kind of thing, so it doesn't look like a problem to us.
Now, most people just use Python.
R is very popular with stats people but new PhD candidates are beginning to write python implementations of R things (sort of how like DataFrames/pandas happened).
People have been saying this since I was an undergraduate.
I'm submitting my tenure packet this year.
> I'm submitting my tenure packet this year.
That doesn't mean it isn't happening. If anything, the fact that the rate of adoption is slow over a long period of time would tell me that there's more stating power here. If suddenly 20% used python that would be a warning sign. If there was .5% year over year for decades.... That's different.
and also "The year R unseats Python."
https://www.datanami.com/2019/08/15/is-python-strangling-r-t...
https://madnight.github.io/githut/#/pull_requests/2014/1
R has halved in percentage since 2018
Sure the syntax is going to seem alien to them, but so would any first encounter with a programming language.
The use case for R simply isn't a traditional programmer. That isn't the target user. Sure if you need an application that might need significant scale you're not going to use R Shiny, but a lot of R work are one-off bespoke analysis projects. Models that do need to be deployed for use at scale in an application take their output parameters from R models and simply implement them in the app. I do this myself, taking coefficients etc, implement a function call in a database and then use the results on the front end.
R is a terrible programming language.
It's not a bad language for invoking code, because for many of those people, they're not taught the concepts behind any language, so it's all semi-arbitrary symbols.
model <- lm(outcome ~ variable1 + variable2 + variable3, data=data)
summary(model)
Isn't any more complex than anything else. And what R does have is a network effect - at this point, for almost any statistical task I've ever encountered, there's R code for it.
I think it’s very similar when it comes to writing code. The more theoretical driven approach, the more structures you’ve got at your disposal to reason about things (eg see Mochizuki’s proof of the abc conjecture).
For example, should you use monad in a small project? Maybe not. But if you are dealing with thousands lines of code every week you may find that if you reimplement some components as monads then suddenly refactoring gets easier and you can extend things more easily without having to do a global search and modify all the occurrences of a certain thing every time you make changes to the type, etc
So ultimately you are offloading cognitive efforts to structures, which are really just constructs to optimize cost-to-transform (though it may increase the cost-to-execute (both computationally as well as mental-visualization/simulation-wise))
So is it worth the effort to work with structures? It really depends on the project you are working on or what you see yourself building in the next 10 years
Depending on how you see your career it’s always good to be a bit more ambitious
Mochizuki finally went ahead and published his work in a journal where he is the editor in chief. So much for peer review.
Hadley Wickham is a very talented developer, and what he's particularly talented at is writing interfaces that are easy for people to use. dplyr, a key part of the tidyverse, is a great example of that. It breaks data tidying up into a few simple steps that can be chained together. It's the descendant of an earlier iteration (plyr) and Hadley learned from that and just kept polishing the interface.
There's a similar story with tidyr. Reshaping data from wide to long is a complex operation. Base R has a reshape() function and using it has given me permanent PTSD. It was impossible to get right until you read the documentation. After you read the docs - still impossible. Hadley wrote the reshape2 package, which improved things a bit. Then there was tidyr::gather() and spread(). Finally, we got tidyr::pivot_wider() and pivot_longer(), and at last I can be reasonably confident of getting the results I need without too many tries.
The tidyverse is hugely popular for just this reason. Without it, I'd probably have abandoned R. I certainly wouldn't dream of teaching it. Ditto Rstudio. Calling these people parasites is absurd.
ggplot2 did a similar thing with plotting. Together these two projects made R more accessible-like and, IMO, seriously damaged the culture around it.
I fully disagree with the disparaging aspersions or even motives that TFA leans towards, though. I can see how this would arise out of a local minimum.
The "beauty" of R has always been its core DataFrame abstraction and the fact that a table is a language primitive — and that's where 90% of the consistency came from (from my outsiders' vantage point).
d <- data.frame(a = 1:3, b = 1:3)
d[4,] <- 1:3
# or
d$c <- NULL
# or
d[c(T,F,T,F), 1] <- 1
So if you want a clean and simple R ecosystem, go support Julia or some alternative like that.
Not that complexity means the programmer was "incompetent", per se, but not sufficiently competent to keep things simple, because as you point out, keeping things simple is really hard.
More than RStudio, it's in Tidyverse users' interests to extend and embrace; even without any plan or malice aforethought, it is simply natural that as Tidyverse users do their own thing, their packages and scripts will rely on the Tidyverse ever more, and their downstream users will thenceforth rely on it whether those users like it or not. Thus the situation where you install.package some harmless looking package and now R is installing 20 or 40 packages from the Tidyverse.
I dub this general phenomenon 'bitcreep': https://www.gwern.net/Holy-wars Somewhat ironically, Hadley Wickham himself has been a bit annoyed by bitcreep caused by Mac/Linux users working against Windows users: https://twitter.com/hadleywickham/status/1280340931657564160
What I find interesting about the quoted excerpt, and the article most broadly, is about how it spends very little time discussing what RStudio's economic interest in propagating the Tidyverse actually is. Because, notably, RStudio doesn't charge for the Tidyverse packages. There's no Tidyverse Enterprise Edition, there's no Tidyverse support plans. They mostly make money off their IDE and server products, and those products don't have a lot of meaningful synergies with the Tidyverse packages; there's few IDE features that integrate directly with the Tidyverse, and those that exist aren't very significant. Meanwhile, nothing about the Tidyverse is written to work better with RStudio than with VS Code using the R language server, or with Emacs, or what have you.
This isn't to say that RStudio is doing this for any reason other than economic interest, but the economic interest here is not in convincing R users to use Tidyverse packages; RStudio makes the same amount of money if you do your plots in ggplot or not, and if you switch to ggplot there's nothing about it pushing you to change your IDE/editor.
So what is the business model for the Tidyverse, then? It's pretty straightforward: the goal of the Tidyverse from the perspective of RStudio is to drive R adoption, mostly at places who are using commercial closed-source alternatives to R, like Matlab, or SAS or SPSS. (You could argue that they're competing for mindshare with Python, too, but RStudio has moved to treating Python the way that post-Azure Microsoft treats Linux, as a part of its product portfolio. RStudio's flagship products, the IDE and the Connect server, both advertise first-class Python support. Whether or not they have achieved this is another question, but they certainly are trying.) Once you have converted people to using R, you can sell them IDEs and servers.
I suspect that the "Base R" people have a hard time grappling with this aspect of the Tidyverse business model, because it implies that Base R is _inherently less popular_ than the Tidyverse, which makes their losing less about the whims of a corporation that they can argue is acting in bad faith, but about the preferences of the R community, which they have no power to gatekeep.
Because the point of the Tidyverse is driving adoption among people who were not previously R users, the Tidyverse is able to win over the majority of the community not by persuading holdouts but simply by growing the community with new Tidyverse supporters. I understand how that can feel threatening to someone who was an R user pre-Tidyverse, but you understand why they don't want to focus their message around shrinking the R community. And their efforts to persuade these new people to switch from the Tidyverse to Base R is undermined by the fact that the best argument for Base R over Tidyverse, the familiarity of Base R to someone who learned Base R to begin with, doesn't apply to them at all. They're fighting a losing war with bad weapons.
This seems to be the root of the argument, and it is a completely bizarre statement.
My rant aside, I did find this next quote from Matloff a better argument for using dplyr:
> mean(Nile[80:100])
With dplyr, you'd subset based on a `filter()`, likely specifying the years to keep. It encourages self-explanatory code. Matloff's vector subscript tells me nothing about why certain elements are kept.
I've accepted that my open source project, htmx, will likely never drive any significant income. It's too simple. Whenever someone starts talking about consulting, I tell them to read the docs and maybe jump on the discord, but that they'll probably figure it out.
In addition to the revenue angle, a lot of developers don't like admitting "that seems too complicated" because it can be interpreted as "I'm not smart enough". Arguing for simplicity often requires a lot of backbone (or disagreeableness, according to taste.)
The blog post is projecting a lot on the situation. It’s a blog. Take it for what it is.
One of the reasons I stopped using Windows was the complexity pretty much forces a support contract for anything other than a toy business.
Funny how distros offering support contracts are taking linux down the same path.
> For the last nine-months I have noticed that the term Tidyverse is being used more regularly to describe what had been the Hadley-verse. And???
Hadley never liked the name 'Hadleyverse' and the first appearance of the `tidyverse` package on CRAN is from 2016.
> RStudio management rode in on the data science wave, raising money from VCs.
RStudio is now a Public Benefit Corporation.
I strongly disagree. I've used both and don't find RStudio clunky. And Jetbrains would need to do more than simply add support for another language to compete. RStudio is more than just and IDE. For one, there's Shiny, which makes it trivial to deploy slick web-based data products to end users. Then there's the fact that RStudio has multiple methods of compiling code and analysis into formal document formats suitable for publishing or presentation.
RStudio isn't just an IDE, it's an entire ecosystem for coding, analysis, app development, and publication. It's not like Jetbrains couldn't make something like that, but it would essentially be a completely different product, not just one more supported language.
[1] https://schleiss.io/measuring-potential-complexity-in-open-s...
"No you can't just use if statements, you need to use a chain of responsibility with many classes cause it's SOLID".
It's not a mystery that one of the author of a popular front-end framework was a JEE consultant, and this popular framework now feels like a solution looking for a problem...
Humans also possess Simplicity Bias. (i.e. we "over-simplify" things). "Simplicity bias is a cognitive bias towards holding views which can be explained by a simple narrative."[2]
Ironically, this very article may exemplify the Simplicity Bias to explain a Complexity Fallacy!!
It seems the ultimate answer is the perennial, "know thyself." Practice self-awareness; and know which Bias you're leaning on, and more important, why you're leaning on it (in light of the goal).
[1]: https://fs.blog/2018/01/complexity-bias/#:~:text=Complexity%....
[2]: https://issuepedia.org/Simplicity_bias#:~:text=Simplicity%20....
Are MySQL and SQLite still the most reliable ones around?
Replace "postresql" with literally any tool and the sentence holds true: redis, mysql, mongodb, cassandra, mssql, HDFS, Kubernetes, React, Vue...
The fact that cottage industries pop up around powerful tooling is not a bad thing. Some things are just complicated, and there shouldn't be such an aversion to those things.
Sadly a lot of the "NoSQL" hype later on was led by people who had no clue about database theory, and proceeded to throw the baby out with the bathwater and build whatever abomination MongoDB is, rather than taking SQL's theoretical basis and building a language on top of it that is actually designed for complex usecases. There were only a few exceptions like say Datomic in Clojure world that did NoSQL right.
So they came up with something that implemented the Relational Model in a simple way, one that would attract new converts. SQL has plenty of warts and defects, but its success at extinguishing every other database query language can't be denied.
The relational model with unions of conjunctive queries at its core is excellent. SQL on the other hand feels like some weird COBOL dialect that gives you an incredibly verbose implementation of it.
With that said, there are a few enterprisey NoSQL databases which are relational yet provide a nicer language to work with than SQL. I mentioned datomic (which also has persistence as its key feature), but there's also TypeDB which lets you use the first class support for recursive queries to support inheritance more easily. And of course there's google's Logica language which compiles to SQL and lets you use datalog syntax and the ability to write reusable queries, though it doesn't support recursion due to the nature of its target.
For example, anyone who's tried to implement SAML SSO by hand or self-hosting a shibboleth server will appreciate the relatively simpler services provided by OneLogin / Okta.
I'm not sure, however, that this is a problem. The freedom of open source to develop without constraints has to come first, so that someone can later take existing code and impose guardrails to simplify.
In other words, freedom itself is at odds with simplicity.
Make a free, open, easy to use app and you'll go bankrupt.
In other words -- if you are free to write code any which way, complexity will result. Or conversely, in order to keep code consistent, someone would need to enforce rules.
I have for quite some time seen the two as being at odds. Free as in free beer undermines free as in liberty by destroying the economic basis for anything other than surveillance or closed business models.
And it wasn't even intentional. The software existed for almost two decades before the commercial company. The commercial company sprang up because the open source authors needed paychecks for all the support they were doing. And also a bunch of companies were selling Sendmail consulting. So they figured people would rather buy consulting from the people who actually make the software.
Sadly, it didn't work out. The company was acquired for a loss to the investors.
From outside this looks like they're holding out on us.
Some tools require google stalking the maintainer in order to synthesize explanations from six different places into a Theory of the System of how the thing is supposed to work. Apache Ant, for a memorable one.
As with many things, you don't know your own opinion of a subject until you have to try to teach it to someone else.
They're doing the fun part (coding) for free. It's well-known that developers don't like writing documentation (esp "properly" which is ill-defined and for some definitions can be a much larger task than writing the code in the first place); in fact, does anyone enjoy writing docs as much as an engineer enjoys writing code?
So this makes total sense to me. As a user of free software, you can choose: a) pay for a commercial package with comes with docs; b) use free software, and pay for the docs in the form of a book (perhaps waiting a few years until lots of people want to do the same); c) use free software and figure it out from breadcrumbs the developer left in whatever forums they use; d) (c) but then step up and write some docs.
Too many people think that free/libre software is supposed to be like commercial software, only better, but it's not the case. It's definitely better in the "libre" way, and of course cheaper in the "price" way, but it's usually not packaged for easy consumption, nor is the developer going to do a bunch of work they don't enjoy for other people who don't want to pay them for it.
"It's free so shut up" only works if being free has no consequences. Software is all social, and we pretend that it isn't. Cutting the legs out from underneath commercial software has consequences. Crowding a headspace with entrants has consequences for the next person considering doing the same thing. Implying you have something worth other people's time to look at has consequences.
You figure out how to solve all of those problems and you can be above reproach. Until then, if you can't do it moderately right, then don't do it at all (or keep it to yourself).
I head the claim and it seems as if it was a line from Microsoft back when they were hostile to open source and a minute after it left their mouth managment promptly set up a meaning to ask "Why aren't we doing that?" shortsightedly to the detriment of their own longterm health.
Open source ethos have been more "not want to deal with that crap" in ethos and a general tendency against wanting to provide support if it doesn't involve bugfixes. Again back when the community were more stereotypically grumpy greybeard telling you to boost your expertise by RTFM than evangalists with sizable blog and/or social media presence depending on the timeframe, or even novices grinding out tutorial websites as a portfolio item of sorts as an extension of "extracurriculars to get into a good college" side activities. I am not calling that cultural change a bad thing, far from it. I am just noting it has far more energy involved in maintenance of a communal efforts compared to less than easily accessible man files whose gap in time between last update and today could rent a car if it was an ordinary human.
Atlassian's Jira is a good day example of what the author's gunning for. A complex, unintuitive, hulk of a product .. with seemingly nonsensical UI updates and archives of out of date documentation.
Amazing for people providing support and training. As a potential user, I've been annoyed by badly set-up instances so many times I'd gladly turn my back on it.
And yes, as a Moodle user (teacher) you notice quickly that these decisions seriously impact usage!
More fundamentally, complexity has more to do with the problem space than the business model.
At this point I would like to read a book about the Confessions of an Open Source Hit Man.
And sure, ```technically''' that's true but there are good reasons that numerics aren't just all polynomials all the time (check out Lloyd Trefethen's 'Approximation Theory and Approximation Practice', for example).
tanh is nice and smooth but its Taylor series doesn't converge outside a small region, and the whole reason you'd use tanh in the first place is because you'll have things outside that region. And tanh is probably the best case for Matloff's argument! ReLU has that corner, and I don't even want to think about approximating softmax with a polynomial.
The blog has comments by, among others, Radford Neal and Kenghagho which I find compelling.
What is closed source anyway? It just means you get binaries. Binaries are not actually obfuscated. You can disassemble them, and there's even some decompilers on the market that will create functionally equivalent C code with auto-generated function and variable names. Nothing stops you from decompiling and reading a closed-source binary... except the complexity and difficulty of understanding it.
Open source systems where the source and installation/admin procedure are absurdly complicated are not actually open... they're just using a different obfuscation technique. If it takes days to install or the source is write-only code, it's no different from closed source software.
This is asked as if there is no correct answer, but there is.
Open Source software is software where the source is made available under an OSI-approved Open Source licence. Closed source software is software which is not Open Source software.
See https://opensource.org/osd
> It just means you get binaries
Sometimes even the binaries are carefully obscured. I imagine it's an uphill battle to dig the binary out of a PS5 game, for instance.
> Binaries are not actually obfuscated.
Some are.
> Nothing stops you from decompiling and reading a closed-source binary... except the complexity and difficulty of understanding it.
And possibly the licence, and possibly laws like the DMCA. Also, even if you succeed in reverse-engineering the code, you may be legally constrained in what you can do. If you improve the software, you probably cannot legally distribute the improved version, although you might get away with distributing a delta.
See also https://www.gnu.org/philosophy/free-sw.en.html
> Open source systems where the source and installation/admin procedure are absurdly complicated are not actually open... they're just using a different obfuscation technique.
I agree artificial complexity (whether in the code or in the software's usability) could be a way for someone to, say, deliberately subvert copyleft licences. An artificially unmanageable codebase would increase the cost of development for the company, though.
As for tricky installation processes, I believe the only way to get the official OpenBSD builds is by buying their DVDs. I don't think most OpenBSD users particularly mind compiling their own system, but I don't know for sure.
> If it takes days to install or the source is write-only code, it's no different from closed source software.
It is different, because the licence allows others to improve the software.
> This is asked as if there is no correct answer, but there is.
> Open Source software is software where the source is made available under an OSI-approved Open Source licence. Closed source software is software which is not Open Source software.
I believe you're missing the point. Parent is making a moral point and your talking about copyright laws and licences stamped by OSI. For one i'm pretty sure parent would much more trust GNU than OSI. Perhaps parent's point would've been clearer if they said 'non-free software' instead of 'closed source'.
And GNU licences define 'source code' as the 'preferred form of modification'. The reason for this being exactly parent's point. As an example, that's one (of the multiple) gripe i have with ml: some 100 lines of python defining a neural network with millions of params doesn't constitute the source code, neither do the trained params. The source is in this case non-textual, it's the training data and the state-dump of the IDE used to manage the training and inspect the model.
> > If it takes days to install or the source is write-only code, it's no different from closed source software.
> It is different, because the licence allows others to improve the software.
But that is pure legalese since it's intractable to modify it. It completely subverts the spirit of the licence, as such it is practicaly and in spirit closed-source.
Extending a bit more the argument, what we want isn't just free software, it's software that's also interoperable, where it's easy to reuse just a part, well engineered (minimaly complex), with well defined scope and powerful/precise interfaces, etc. That software is golden, the rest will just disappear in 20 years.
I agree it's important that our understanding of something like Free Software remain relevant and practical. At the same time though I think quality and freedom can generally be pretty well separated. If you release poor quality code under a Free and Open Source licence, we generally agree it still counts as Free and Open Source, despite that it's hard to work with. I think that's as it should be, although deliberate subversion might be an exception.
Are there any examples of this actually happening, though?
I imagine the best way to create obfuscated code is to use an obfuscator program, but the result of doing so is not considered to be source code, for our purposes (the preferred form of modification, as you rightly noted). If someone wants to deliver obfuscated source code and pass it off as Free and Open Source Software, they would need to program it manually, which would be severely punishing to their development work.
> what we want isn't just free software, it's software that's also interoperable, where it's easy to reuse just a part, well engineered (minimaly complex), with well defined scope and powerful/precise interfaces, etc.
That strikes me as clearly going too far. Technical excellence is not among the four conditions that define Free Software, neither should it be.
It also has absurd consequences:
• If you write a program in a language which then falls out of favour, do we say that it was Free Software at the time but no longer? (Software freedom is now defined in such a way that it is a function of which programming languages are currently in vogue.)
• If you are an outstanding programmer, you are better able to cope with poor quality code and with code in obscure languages, broadening what software you are able to work with. (Software freedom is now defined in such a way that it is a function of the individual programmer's skill.)
• If you develop a new algorithm and make the source code of your implementation available under a Free and Open Source licence, before you publish a paper explaining your new algorithm, the implementation is considered non-Free prior to the paper's publication, but Free afterward. (Software freedom is now defined in such a way that it is a function of available documentation and literature.)
Similar points might be made on the grounds of what hardware is currently available (does the non-availability of an emulator impact whether software counts as Free?), and what institutional resources are currently available (should laser-cutting software be considered non-Free by most, but Free by the relevant industries?). As an example, NASA's old launch programs [0] are at the intersection of these two concerns.
[0] https://qz.com/726338/the-code-that-took-america-to-the-moon...
It's not good at solving problems in the best possible way according to everyone. Centralisation is great at solving problems in the best possible way according to a single person or circle of people.
It's good at solving problems so that the majority of people willing to pledge resources are happy.
1) javascript and jquery
2) python and numpy (and the scientific stack built atop it)
In neither case was the primary incentive, at first anyway, a desire to make money by complexity lock-in. I think there is a real cost to the complexity of having what is, essentially, a whole second language (with its own syntax and ways of doing things) built on top of the original language. But, I think this kind of thing can and does happen regardless of the financial incentives involved.
I often joke that Angular(or specifically any app written in it) is a job program for front-end developers.
The architecture of this framework is so byzantine that you can spend _years_ working with it and still keep discovering new, often insufficiently documented features.
Or it just grew that way. Which is the most likely explanation.
They are actually fighting for survival I would say.
Most of the people using R and RStudio are the people who only know R and RStudio.
RStudio's interface feels like it is from the early 90s, and RStudio is impractical and messy to use. Frankly, it is _bad_.
R, as a language does only very few amounts of things well. And when some people's use cases that are limited to that, uses it.
Data Science wave riders dominantly learn Python (IMO, as they should). Think >90%.
There are plenty Data Science Misguiders (influencers) who, just from shallow knowledge, tell gullible learners to learn R OR Python. And still almost everybody chooses the second.
Some people I know who uses R amd RStudio uses them as a simple Dashboard, dataviz app, or those who uses it for other reasons write R using Jupyter, VSCode, or good ol' vim.
R thrives on the backs of lesser skilled developers amd domain experts using R for some reason.
Some argue that the tooling of R projects is better than Python's, but it does so less. A penny is of course more reliable than a train.
I tried R but it felt weird, and the community felt more elitist.