Hacker News Reader

Comment by peatmoss | Hacker News Reader

The Tidyverse is a true gem in modern programming that deals with data.

This release feels like a nice incremental improvement rather than a revolution. Adding lubridate as a core package will save me a few keystrokes. I also did not know about the conflicts package, but will certainly be using it in the future.

I'm a bit surprised that other "better" programming languages haven't done more to replicate the Tidyverse. The Tidyverse is a masterclass in metaprogramming, and has a pretty big following. While R has some warts as a language, I'm not sure I've seen anything comparable to Tidyverse in terms of scope and influence. I see Julia trying with the Dataframes ecosystem, but it's got a ways to go before that feels cohesive like Tidyverse.

If I could wave a magic wand, I'd get Hadley and the RStudio crew really excited about Julia or Racket. R does have warts, and I dream of the day when something better is as good.

View on HN

I don't know if it's impressive or worrying regarding how much Posit/RStudio as a company have completely taken over a programming language.

I have colleagues who write R code all day, and they don't know the difference between RStudio the IDE and R the language. They can't perform basic tasks such as defining a function, or adding a column without the tidyverse.

I see that same thing all the time. RStudio = R, and they have no concept of the language without the IDE. Which is fine, I guess? They are not professional programmers, but people with a job to do.

It is just frustrating to me when they RShiny all the things, and I have to pickup the pieces because the spaghetti has collapsed under its own weight.

I see it a little bit more like a meritocracy in that space that RStudio has largely won. Microsoft has certainly made an attempt to become the face of R. Remember when they purchased Revolution Analytics and started baking it into everything? Or how about Oracle making a play on Graalvm with "FastR"?

RStudio has somewhat indefatigably looked for pain points that R users had back in the olden days. And yes, I remember R back when lattice plots were the hottest game in town. Not only has the Tidyverse made R infinitely more pleasant to work with, the community around Tidyverse has made the R community infinitely less insufferable for newcomers. The obtuse responses to beginners on the old R mailing lists were legendary.

My preferred R IDE is emacs, and I'm not upset with what RStudio has done. When I interact with devtools, I don't feel that I'm wanting for functionality. This is in spite of the fact that RStudio has heavily integrated devtools functionality into their IDE, and could arguably have made devtools less easy to work with using generic tooling.

I understand the mediocrity of many RShiny applications in terms of how they are written. But, another way of looking at all those dashboards and interactive applications, is that they are AMAZING prototypes written by scientists, grad students, journalists, etc., who would have never been able to build them some other way. If we forced everyone to write these apps in Rust, of course the average quality would be higher, but we'd also have way fewer of them. Democratization of technology means that its users will be more... medium.

>...who would have never been able to build them some other way...

It has definitely become the best available MVP generator. I just wish I liked R more. Python still have nothing even close to how easily a novice can spin up a Shiny application.

I'm not sure how usable it is, but Shiny for Python exists: https://shiny.rstudio.com/py/

I think that package is still pretty early days, but does show promise. I was speaking less to the code and more to the ecosystem.

If I tell someone they should use Streamlit/Dash/Shiny for Python, how do they get started? The beginner onboarding experience is horrible for someone without an existing programming background. Download Python, maybe instruct them to setup a "virtual environment", use this "pip" thing to install these packages, point them at a way to edit code, and try to give them a way to launch it. God help them if want to share their work. Huge amount of existing resources assume Linux by default or that people have any idea how to use Docker.

While I hate...basically everything about the R ecosystem, the beginner experience is leagues better. Download RStudio. It will setup R, install packages, and has a big button that will launch the Shiny application. Straightforward guidance on how to get it hosted on an existing provider.

Julia seems to be doing a better job catching up to R in this space than Python. I haven't used it personally, but the demos of Genie Framework are impressive: https://github.com/GenieFramework/Genie.jl / https://genieframework.com/

I write in R for much of my work days, and perhaps because I learned when the tidyverse was in its infancy (or before) I don't use much at all of its packages besides ggplot2. With the base r pipe operator I use zero dplyr and find most of its functionality has long existed in base R with functions like transform(). lubridate has a few good convenience functions (that ultimately I could do without), and I do like plotting in ggplot2 - in fact the latter is the only place I reckon there has been too much of a takeover because most plotting development effort I imagine is based on the grid and ggplot2 framework now. I do like tidyr::pivot_longer(), although it wouldn't change much if I had to use the less intuitive base R versions for dataframe manipulation.

Your colleagues are probably a symptom of how low the barrier to entry has been made by the tidyverse moreso than indicative of a total capture of the language.

I like the tidyverse in general, but I have to admit I've never gotten the hang of tibbles. I use dataframes because I like row names and pretty much every non-tidyverse package uses dataframes and row names too. I really don't understand why row names (a basic feature of R) are viewed with such disdain by the ecosystem and can't be set using tidyverse functions like read_tsv. I generally end up converting them to dataframes after using them.

Most R programmers are scientists or statisticians though. They aren't programmers in a professional sense. They are using a tool to get things done for their real work, and if RStudio helps them, so much the better.

> This release feels like a nice incremental improvement rather than a revolution.

Look at the big picture, not simply the delta from the prior .x.y release.

The Tidyverse feels like a product that evolves over time, rather than a traditional approach that jumps versions. Major changes sometimes come in seemingly minor releases. I feel that 2.0 simply means the evolution has gone on enough to put a retrospective marker in the sand.

For something geared more to data science and exploration, rather than production use, I’m good with the evolutionary approach.

I agree 100% that this is a good thing. I had intended this as a compliment that Tidyverse had achieved this level of maturity, but failed in my under-caffeinated state. Had Tidyverse 2.0.0. been a revolution, I'd have been worried.

Well said, I completely agree with you!

>I'm a bit surprised that other "better" programming languages haven't done more to replicate the Tidyverse

Python has a very weak metaprogramming story so it is hard if not impossible to replicate something like Tidyverse in that.

If Ruby had succeeded in data science instead of Python it would almost certainly be with Tidyverse-like abstractions.

“Tidier.jl is a 100% Julia implementation of the R tidyverse mini-language in Julia.”

https://github.com/TidierOrg/Tidier.jl

This illustrates the point perfectly. Julia is attempting this and has a beachhead with Dataframes.jl. Confusingly though, Tidier.jl isn't really analogous to R's Tidyverse. It's more like one of a handful of meta-packages around Dataframes.jl.

Then there are Grammar of Graphics (ggplot was Tidyverse's first star) style plotting libraries that Julia has been building. I'm probably most excited about Algebra of Graphics (https://github.com/MakieOrg/AlgebraOfGraphics.jl/) as part of the Makie Plots ecosystem. It does still feel a bit like Julia community can't decide between following Matplotlib or R's Grid/Ggplot approach.

The seeds of a Tidyverse for Julia are there, but it'll take some time to achieve the consistency and maturity of the original Tidyverse.

I think DF.jl works remarkably well as a "fits in RAM" dataframe backend, but I think it just lacks in usability and integratedness with the wider Julia ecosystem. Or rather, the wider ecosystem isn't as mature in key data analysis areas.

In particular, as you mention, plotting is one of the evolving parts of the ecosystem. Plots.jl is fine, Makie is powerful but very DIY, and AoG is slick but unwieldy. ggplot2 is far from perfect, but it works so well due to its maturity and integration with the rest of the Tidyverse.

In my ideal world, there would be a DataFrames.jl wrapper to provide nice (not just nicer like the two DFM.jl packages) syntax, and a powerful high-level plotting package (Makie is powerful but syntax is low level, Plots is mid on both) which is heavily integrated with the wrapper package.

Admittedly, I'm not a data scientist (anymore) so I don't follow the new developments in the dataviz scene much. If something like this exists then I would love to find it.

I wonder what my ideal syntax would look like anyway. Maybe something close to Tidyverse but with symbols as column names `:col_name` for ambiguity reasons.

I'd agree with all of this. It's not so much the presence of ecosystem, it's the maturity of the ecosystem and knowing how to navigate it. Lots of competing approaches in a relatively small community makes me a little nervous.

If people are curious, DataFrames.jl has done a fair bit of work in consolidating a list of other packages that complement DataFrames.jl:

https://dataframes.juliadata.org/latest/#DataFrames.jl-and-t...

I do have to say that DataFrames.jl itself is a pretty impressive bit of work, and Bogumił Kamiński deserves no small amount of credit. If Julia ends up stealing significant mindshare from R / Python data science communities it'll be because of of this work.

Also for the Julia curious, I'll say that I find the ergonomics and sensibilities of the DataFrames.jl ecosystem much more in line with R and the Tidyverse than Python/Pandas.

Bogumil is a truly outstanding member of the community and DataFrames.jl is an impressive, versatile package.

From my perspective, however, DataFrames.jl's power is what makes it quite unergonomic for me. As an example, take the `args => transformations => result` syntax for doing pretty much anything in DataFrames. It versatile, but the lack of rank polymorphism in Julia i.e. broadcasting/mapping has to be explicit (which is usually a good thing given that type polymorphism is Julia's whole schtick) means that the transformation syntax feels cumbersome.

It's not that I want everything rowwise by default, an option provided by DataFramesMacros.jl, it's that I want things to be rank polymorphic when it makes sense. Base R got this right, hell S got this right, and so the Tidyverse inherited it and it makes the package so much more ergonomic than it would otherwise be.

I cannot overstate how impressive DataFrames.jl is, but I have to caveat this with "but I really try to avoid using it if possible". It's a shame, but I just think R's laissez-faire hackability, which in many cases results in spaghetti code, works really well in the tabular programming world where ergonomics are king and performance is easy.

> In my ideal world, there would be a DataFrames.jl wrapper to provide nice (not just nicer like the two DFM.jl packages) syntax

Have you come across [Tidier.jl](https://github.com/TidierOrg/Tidier.jl) yet? It's a relatively recent package (to my understanding), but developing at a pretty rapid pace, and tries to be more ergonomic with its syntax.

Reply on news.ycombinator.com