johnmyleswhite

There's work being done to wrap Dato's SFrames library, which would solve that problem.

I think you may have misinterpreted that post. Look at the table under "Comparing Performance in R and Julia" again.

The primary use case for Julia's coming array slices is to provide views into an array that you always intended to keep alive anyway.

Iterating over the columns of an in-memory matrix, for example, is the use case that this change is meant to speed up.

That issue is largely resolved in 0.4. Some functions are still a little slow, but most packages will now load much more quickly.

If you'd like to know canonical programming style, you should read the code in Julia Base. Learning how to write good Julia is somewhat difficult because the language occupies an unfamiliar position between traditional static languages and traditional dynamic languages.

Can you provide some concrete examples of the functionality you're looking for? What exactly does "quick-and-dirty" code look like to you?

Probably worth noting that the GitHub issue in question relates to work being done on Julia 0.5, a future release that is not yet ready to ship and won't ship for several months.

Within a few weeks, Julia 0.4 will be shipping, which is not focused on changes to arrays.

I've been doing a lot of work behind the scenes there, but it's still going to take a while to offer a mature product. What we really need are more qualified engineers to speed up the process, but there's a vanishingly small group of people who are interested in doing eng work on statistics libraries.

If you're interested in this topic, you should watch this talk by Steven G. Johnson: https://www.youtube.com/watch?v=jhlVHoeB05A

tl;dr of the talk: Syntax has little to no effect on how a language performs. What distinguishes Julia from Python is that Julia's semantics were designed to be amenable to type inference. The results of type inference allow Julia's compiler to generate very efficient machine code.

FWIW, I think the DataFrames package and its dependencies have consistently operated at the boundaries of what we know how to do efficiently in Julia. The package has had lackluster performance in many contexts primarily because it adopted many idioms from R and Python that were sharply at odds with Julia's type inference system. We're starting to clear those problems up, but there are still lots of unsolved challenges we need to resolve.

If you have any ideas about how we should modify the basic data types and functions defined in DataFrames, those ideas would go a long way to making Julia a better language.

GNU's assembler is implemented in C.

If you're compiling Julia from source, you should be able to do something like write a base/userimg.jl file that contains this kind of text:

Base.require("Distributions.jl")

Base.require("Optim.jl")

Base.require("DataFrames.jl")

Base.require("Gadfly.jl”)

You'll need to create a new build after adding this userimg.jl file. Also be aware that changes to these packages will stop showing up when you type "using Gadfly" since you'll always access your precompiled version.

Have you tried the userimg.jl hack? It's _very_ hacky, but it will convince you that precompiling packages is viable in the short-term.

I'm not sure. I think, for many users, SQLite is sufficient, which is why I'll be using it for prototype work in the future. Trying to add things like arrays seems like a lot of work, since you also need to add things like HQL's EXPLODE at the same time.

To echo what I said in the post, I think using Nullable as the representation of missing values is the right strategy. I think it achieves sufficiently strong performance while also providing a lot of other desirable features.

As for other things that we've done well, I think the Distributions package is one of the best designed interfaces to probability distributions I've ever seen in any programming language. I don't know if Go supports value types, but the existence of value types in Julia allows each distribution to have its own distinct type without having to pay any cost for maintaining object identity. When combined with Julia's ability to do multiple dispatch, the Distributions package allows one to encode analytic knowledge about probability distributions directly into the type system. And existence of parametric types lets you talk about things like location-scale families in a natural way.

I'd also encourage you to look at the optimization libraries that have been built for Julia. The language makes it easy to do automatic differentiation, which is probably the most under-utilized tool in all of scientific computing. In general, I think there's really high quality work being done on optimization -- which I see as essential for enabling ad hoc statistical modeling.

I am definitely going to use embedded SQLite in some of future prototypes. The only problem is that SQLite doesn't, to my knowledge, support all of the features I'd like Julia to support -- including columns that contain arrays, dictionaries and even complete tables. But, that said, it supports >90% of the features I'd be interested in.

I'd encourage you to read the post again, since it's primarily focused on describing design patterns that allow one to support missing values without taking a performance hit.

We will probably be pushing people very heavily towards always using specialized arrays for working with nullable objects.

I'm a little surprised by your comment about sufficiently smart compilers, since Julia seems to have taken the exact opposite approach to compiler design. The language is systematically designed to require little intelligence from the compiler. Being able to safely drop tag checks inside of a function's body seems like a fairly simple part of the language's design, since the type of all inputs to a function and all function-local variables are, in general, easily knowable as invariants before a function call even begins evaluating.

We'll keep re-evaluating this over time, but I'm currently not sure it's that serious a problem. For many numerically intensive computations, you'll want to move over to a pure array of 64-bit floating point values before the long-running computation begins, at which point you can adopt NaN == NA semantics if appropriate. I believe the solution I outlined in my updated comment above may already describe the buddy bitmap solution you're referring to. It certainly describes a solution that I would refer to as a struct-of-arrays.

I didn't mean to imply I thought you would guess the wrong answer, but rather that most R users do not understand delayed evaluation well enough to understand how it will impact their code. Sorry if my comment seemed rude.

Thankfully, the solution paths you're enumerating aren't the only viable solutions to the problem of representing missing values. Because Julia has value types (called immutables) and because Julia can compile functions based on their input arguments' types so that the functions don't need to do any run-time tag checking, a Nullable{Float64} value can be stored using at most 16 bytes -- 8 bytes for a Boolean flag indicating whether a value is present or not and 8 bytes for the actual 64-bit floating point value. The latter 8 bytes can store arbitrary bits if the Boolean flag is set to indicate that the Float64 value is missing.

If you're interested in the details of how Nullable{T} works, you can start with http://github.com/JuliaLang/julia/pull/8152 and then read through Julia's compiler code to understand how immutables are represented in memory and how functions that employ them are compiled to machine code.

UPDATE: Looking back at my answer, I should make clear that, when you want to work with SIMD or GPU operations, one can (and we will) use a representation of arrays of Nullable objects that places the values and the Boolean flags into two separate arrays. The values array is then identical to a normal 64-bit floating point array and allows all of the same operations. Setting the Boolean flags appropriately requires a bit more work. (How much work is strongly dependent on the exact computation being performed.)

That's true if Rational means Rational{BigInt}, but the default in Julia is for Rational to mean Rational{Int}, which can potentially produce less reliable answers that using Float64.

My experience asking people about that snippet is that very few people can correctly predict the results of that function without actually executing the code. What interests me about the question is that many people's mental model of how R programs are evaluated is often too vague to enable them to produce sharp predictions about program behavior prior to executing code.

I think it would be great to develop support for generic sets in a separate package that could eventually be merged into Distributions. I'm not sure that I think all of the issues you've linked to are related to that problem, though. For example, the issue about using Vector{Rational} as an input to the Categorical distribution type's constructor is a relatively minor issue since there are very few cases in which you'd want to evaluate the PDF of a Categorical variable using rational numbers rather than floating-point numbers. As such, it's not clear that you'd want to have a Categorical{Rational} type that's separate from the current Categorical type (which could be called the Categorical{Float64} type).

A fun test to see if R users understand how R works is to ask them to predict the behavior of the following snippet of code:

foo <- function(a, b) { if (a == 0) { return(1) } else { return(2) } }

foo(0, stop("Error!"))

foo(1, stop("Error!"))

I fully share that viewpoint, but there are many R users who believe that R's idiosyncratic evaluation rules are essential to writing "readable" code.

I often say this, but I think if you have doubts about whether you should learn Julia or Python, then you should certainly learn Python.

Looks like the growth curve starts before then: https://books.google.com/ngrams/graph?content=software+ecosy...

Right now, DataFrames is handicapped by problems with how DataArrays was designed. With some revisions coming in Julia 0.4, it should be possible to fix a lot of the worst problems (esp. performance) with DataFrames.

Yeah, it's a huge bummer that this leaked before it could be set up for a heavy load.

Resolving this issue is one of the main goals for the 0.4 release. There is already a way to get much better performance for frequently used packages, but it involves a level of tinkering that's prohibitive for most users.