No, shut up. What statistical programming languages can learn from Dropbox.

  def summary(data, key=itemgetter(0), value=itemgetter(1)):
    for k, group in groupby(data, key):
      yield (k, sum(value(row) for row in group))

However, I don't think the programming concepts one needs to understand to make effective use of a well designed Python library are too much to ask. I've only dabbled in R, but when I did, it required me to exercise my general programming knowledge to understand list, matrices and functions. I think the author is also falling in the trap of what is obvious to him is obvious to everyone. I'm actually not sure of what the SAS code is doing, and much prefer the Python.

Languages that have good support for optional arguments, such as Python and Lisp, also make it possible to create APIs that are elegant and concise for experts but extremely intimidating for beginners. It may be more elegant to have a single function with a slew of optional arguments, and an experienced user may be able to accomplish any task quite concisely by specifying a few arguments, but a beginner would be better served by a handful of specific functions with specific names. API writers should consider providing those functions as simple wrappers to the general API, in order to provide a simpler learning curve for users who might never need more complex functionality. Examining how those wrapper functions are implemented can help intermediate users figure out the general API, too.

They want to get at nicely organized data easily from inside of Excel. Excel is a toolbox that they already know from which they can do their own pivot tables and graphs. And they'd prefer to do that because then they can just do it instead of less efficiently having someone else do it for them.

They want it to arrive nicely aggregated and organized, since Excel is not very good at that. But they are more than happy to do the pretty reports themselves. Just get them the data.

See http://bentilly.blogspot.com/2009/12/design-of-reporting-sys... for a more detailed description of one set of experiences that taught me that.

I think the truth is business users don't want to work with you - they just want their data relationships to be discovered in a simple and intuitive way. If your data crawling is sufficiently good, maybe you can do that.

I understand the point the article is making, but I feel rather strongly that "(y)ou’ll want a third thing – to read in and parse data" translates to the person that builds a tool that does that nicely and automatically creates pivot tables, graphs, and other dashboard-y things will probably have to hire a team to shovel the money off so he/she can breathe.

I suppose it's an ok example to talk about this re: statistical programming languages, but my own experience in the three requirements that preface the whole discussion (pivot tables, graphs, data parsing) are a big example of something just screaming for a new solution, not a new prog language . . .

  import numpy as np
  import tabular
  
  # CSV with Region, City and Sales columns
  data = tabular.tabarray(SVfile = 'data.csv')
  
  # Calculate the total sales within each region
  summary = data.aggregate(On = ['Region'], AggFuncDict = {'Sales':np.sum}, AggFunc = len)
  summary.saveSV('summary.csv')

Managers and other folks who need to make pivot tables, graphs and related things without programming have a great tool to do that: Excel. For these people, Excel is Dropbox.

That's what the cookbook recipe provides, a function called summary() that makes a pivot table. Problem solved :-)

> I should be clear that my complaint is with > Python rather than the code as such.

There are plenty of ways to write the summary() function with plain, straight-forward Python code that doesn't use generators, itertools, or any other advanced feature.

So, why does the recipe author use itertools? It is because they provide a way to get C speed without having to write extension modules. Had the author used map() instead of a generator expression, the inner-loop would run entirely at C speed (with no trips around the Python eval-loop):

  for pivot_value, row in groupby(data, key):
      yield k, sum(map(value, group))

I use those every day.

The systems that I've seen/evaluated are Needlebase, Birst and Spotfire. None of them are particularly cheap, but if you're in a business where real-time access to data would help your team make better decisions, they could be very valuable.

Also, leave Python alone!

  # group by Species, can be multivalued see ?by
  # sum(Sepal.Length, Sepal.Width)
  # mean(Petal.Length, Petal.Width)
  by(data = iris, INDICES = iris$Species, FUN = function(x) {y <- colSums(x[,c(1:2)]); z <- mean(x[,c(3:4)]); result <- list(y,z); result})

"Dropbox uses Python on the client-side and server side as well. This talk will give an overview of the first two years of Dropbox, the team formation, our early guiding principles and philosophies, what worked for us and what we learned while building the company and engineering infrastructure. It will also cover why Python was essential to the success of the project and the rough edges we had to overcome to make it our long term programming environment and runtime."

http://us.pycon.org/2011/blog/2011/02/07/pycon-2011-announci...

As a Statistician, I used to use SAS, Stata, R, Excel and of course SQL to extract data but for the purposes of pretty, pretty charts, Tableau is king.