StructuredQueries.jl – A generic data manipulation framework | Hacker News Reader

StructuredQueries.jl – A generic data manipulation framework

julialang.org

So, it seems reasonably dplyrish and I like that there's a pipe-forward operator available (and I do like the look of the F#/Erlang style |> more than I do the %>% one R gets).

This seems like it's written for someone who has a bit more of familiarity with Julia than I possess, though, so is anyone able to give this some context? Why does this need to be implemented with the @query macro rather than just being implemented with functions like dplyr is?

You can use functions in R because R is a crazy language that lets functions access the expressions which are passed as parameters and not just the values of the arguments.

Non-standard evaluation is one of the best features of R.

You mean like this?

http://adv-r.had.co.nz/Computing-on-the-language.html#nse

So, this use-case would seem to point out one advantage of being a "crazy language" in this case. What are the downsides of NSE?

Most people don't know that R has a OO and a Functional side to it. After I learned Racket my R programming changed. After hunting down why I found out that the authors of R wanted to add scheme into the S language when they created R. So you can do some very functional style writing that looks very elegant.

It's a cool feature which enables some really nice APIs, but it also has the potential to break your mental model when reading code. For example:

    x <- f(a)
    y <- g(x, z)

    y <- g(f(a), z)

It would be extremely unexpected if the result `y` was different for these two expressions, but you can't know for sure without reading `f` – since it could be using NSE. This applies to any innocent-looking function in any library you use – imagine trying to debug an issue caused by something like this.

It's also awful for performance. Allowing functions to peek at their arguments at any time is one of the things that makes R virtually impossible to optimise effectively.

Of course, libraries like dplyr also show that it can be used to great effect in the right hands.

I think this is a big part of why R is such a success amongst data analysts.

A lot of R libraries have a really good human interface which you can play around with and it will successfully DWIM a lot of the time.

Once you start trying to drive R in a more automatic way with your own programs, though, it starts to get really messy. I wouldn't want to make even a medium-sized program in it (10,000 lines would be too, for example). Then it's time to switch to Python with numpy.

I wonder if Julia will be able to cover both types of program effectively. There's definitely a niche for it if it can.

I don't know if this is a _general_ downside of NSE, but in the specific context of dplyr, I've found that the focus on NSE has made it a pain to do things in a standard-evaluation way, like if you're writing general functions that use dplyr under the hood.

Dplyr provides SE versions for most all functions, but you frequently need `lazyeval::interp` when building expressions that do work (e.g. for `mutate_`): https://cran.r-project.org/web/packages/dplyr/vignettes/nse....

Notice that document avoids full examples of se-dplyr + lazyeval; I suspect this is purposeful because it tends to look horrid compared to the usual NSE approach. That said, dplyr is still an amazingly productive paradigm shift and I'll keep using it.

Yes, they're isn't really any dedicated syntax in R for manipulating unevaluated expressions, which makes it difficult to write comprehensible metaprogramming code.

Isn't this a feature of Lisp-alikes in general?

In most lisp-like languages, there is a distinction between functions (standard evaluation) and macros (non-standard evaluation). A given callable is either one or the other. In R, every function is both, since it can access both the evaluated and unevaluated forms of each of its arguments.

Hi! I'm the author of the blog post and maintainer of the discussed package.

As CurtHagenlocher notes, in R functions can access passed expressions without evaluating them. So, even though the name `sepal_length` may not be bound to any object in the scope in which `select(sepal_length)` is called, one can use R's non-standard evaluation to process the expression `sepal_length` without trying to resolve the name.

Julia does not have non-standard evaluation, so `select(sepal_length)` necessarily attempts to resolve `sepal_length`, which is a problem if `sepal_length` is not a valid name. Using a macro to an extent emulates the effect of R's non-standard evaluation, at least when it comes to being able to name columns as arguments to manipulation verbs without having to symbolicate them or wrap them in quotes.

The advantages go beyond name resolution. One could resign oneself to symbolicating everywhere, i.e. writing `:sepal_length` to refer to the appropriate column. But then one starts to run into issues when one tries to handle expressions involving function calls. For instance, suppose you want to implement `filter` as an actual function. Well, you could ask users to invoke `filter(iris, :sepal_length > .5)`. This is fine so far as name resolution goes. But the argument `:sepal_length > .5` will throw a `MethodError` when evaluated, because there's no method for comparing a `Symbol` object to a `Float64` object. Using a macro-based regime allows us to access directly the AST representing `sepal_length > .5` and generate a lambda that mirrors the predicate's structure.

One of the main points of the post, which perhaps gets lost in the verbiage, is that, while we're using macros to allow for unqualified (i.e., un-symbolicated, un-stringed) mention of column names in the context of manipulation verb arguments, we can perform a number of other services (such as "automatic" lifting over `Nullable` arguments) and optimizations (such as having the resultant lambda take a tuple argument to avoid splatting, which is costly).

Was this helpful?

So, that makes sense and I learned something, thank you. I'm not sure I know what "lifting" means still, though.

"Lifting" in this context essentially refers to extending a function defined on certain domain of values, say X, to a domain X' which "includes" (air quotes because see below) all the values in X and some representation of a missing value from the domain X. The extended function should derive its behavior from the original function in some systematic way.

More concretely: suppose we have functionality `f` defined over a domain of objects of type `T` -- that is, we're given a method `f(x::T)`. In this case, the domain X is the set of all values of type `T`. The extended set X' is the set of all values of type `Nullable{T}`, where `Nullable{T}` is a specialized container type that can contain precisely zero or one values (an empty `Nullable{T}` object represents a missing value of type `T`, at least for our purposes). I quoted "includes" above because technically the domain of values of type `Nullable{T}` doesn't strictly include values of type `T` (the original domain X), but it includes a set of values that is "isomorphic" in some loose sense to this X. Lifting `f` over `Nullable{T}` arguments means systematically deriving some behavior for `f(x::Nullable{T})` from the original behavior `f(x::T)`. The standard semantics for this are to return `Nullable(f(x.value))` if `x` is null and return an empty `Nullable{U}` if `x` is null, where `U` depends somehow on `f` and `T`.

It's worth noting that the solution to lifting adopted in SQ does not automatically give lifted behavior to `f(x::Nullable{T})` when the latter is invoked outside of a querying macro.

So when called with null for a required parameter, return a null result directly instead of calling the lifted function.

So, if x is NULL return NULL, otherwise f(x)?

In short, yep. For most f. Sometimes we want to do something different, as in three-valued logic.

Okay, thanks for explaining all of this.

It could be that I'm misunderstanding the problem, but I don't think that, in general, makes a lot of sense to define functions acting on NA values. What you really want are functions acting on arrays that may be missing some values, but not necessarily functions that can act on those missing values. This is because the NA values are an effect of the data tabulation, not some deeper mathematical necessity. In that sense, wouldn't it be much easier to endow every array with another list with the places where values are not available and define procedures that use other underlying functions to compute the results while filtering out the NAs? For instance, you could have two "mean" functions, one that filters NAs and then returns the mean of the rest, and another that returns the mean of the whole array and if there is an NA it just returns NA. Both functions could use the mean(x::Number) function but just wrap it in a different way. Or is this missing the point completely?

Reply on news.ycombinator.com