christopheraden

The key thing to note is that the parameter being estimated is a fixed (but unknown) quantity. Unlike in Bayesian inference, where we assume the parameter (or our belief in it) is random. The distinction is important. We want inference about the parameter, yet because the interval is the random quantity with confidence intervals, we cannot assign probability statements to the parameter.

As a completely fabricated example, suppose the true proportion in a coin flip experiment is 40%. If my confidence interval is [.45, .65], what's the probability that .40 is in [.45, .65]? It's 1. The probability that a fixed, but unknown parameter will lie in any confidence interval will be either 0 (it isn't in the interval) or 1 (it is). The _proportion_ of times the interval contains the true parameter is the level of confidence (95%).

To your always-positive example, that procedure is not particularly weird. There's always a balancing act with CI's about length and confidence level (otherwise, I could choose all reals as my interval and get 100% confidence level). That your [0, inf) interval has 100% coverage means that you could probably shrink that interval so that it has finite upper bound without losing more than 5% confidence. Hard to say without a specific distribution in mind or mild assumptions, but an application of either Markov's or Chebyshev's Inequality would allow you to make really loose bounds with only relatively minor assumptions.

I've been with Anthem since going back to the UC system.

Is there any way to check if I'm affected by the breach? University of California has not made an official statement regarding the breach whatsoever.

I'm looking for something similar to the way you could enter your email address and figure out if your Adobe account was hacked.

I'd be interested. Any tool that checks correctness is one less thing I have to write before I begin my analyses.

UC Davis student here. To be fair, ECS10 is not a class that CS majors usually take--it's primarily an elective, and taken by non-majors as a way of satisfying science and engineering general elective credit. The CS majors start with ECS30, which uses C; ECS10 isn't even a required class for the Computer Science programs.

As for whether or not they are computer schools, that criterion was determined through USN&WR, as mentioned in the article, which is where your criticism of not being a computer school should lie. Davis is listed at #34 though, so there may be a sharp decline after the first 10, first 15, etc...

A lot of these "facts" seem like they came from a Quora article from several months ago. Even the wording is similar on some of these, including "If you ask someone a question and they only partially answer, just wait. If you stay silent and keep an eye contact they will continue talking" which is word-for-word what was posted there.

I mention this because some of these seem dubious, and I'd have loved sources. If they came from potentially unsourced Quora answers, the dubiousness just gets passed along.

Link: http://www.quora.com/Tips-and-Hacks-for-Everyday-Life/What-c...

There's some additional information you'd need to determine this. Suppose, for the sake of argument, I only have one feature, X. Pretend I extend it into two dimensions by simply replicating the feature. In two dimensions, (X,X) forms a perfect straight line, which should make it clear that using an additional dimension didn't gain you anything.

Testing whether or not to include additional terms requires an understanding of the distribution of the response, as well as the amount of collinearity with the features (how similar the features are). There are some ways to do this in statistics, but this is more of something they do in inference as opposed to prediction.

Heuristically, the most common way is just to look at the cross-validated classification error and compare it with and without a feature (or set of features) in question. Asking about the distribution of the cross-validated classification rate is an interesting statistical question, though!

Derivatives require continuity. It would be sufficient to simply look at which number of dimensions gave you the best cross-validated classification rate.

I work in big insurance, too. How did you handle porting your legacy code into (R|Python)?

We had tens of thousands of lines of SAS code from over the course of 10 years (macros, clinical programs, reporting functionality) of SAS programming that the higher-ups never saw the benefit of switching to Python or R to be feasible, especially with the Biostats team working on existing projects.

My first question was how CVS-Caremark, a publicly traded company would be allowed to do this, considering it would hurt their bottom line. I didn't think ethics were a reasonable excuse for knowingly hampering profits, unless they felt there was some long-term benefit to their business of not selling cigarettes. Fiduciary duty trumps morals.

Does it have anything to do with the fact that CVS-Caremark is one of the largest PBMs in the US (second only to ExpressScripts, actually)?

There's more at play here than just a moral action.

False dichotomy about Excel vs. DSLs like R. "90% of the world is using Excel and similar tools to analyze and visualize their data. [...] What’s the next step-up from the spreadsheet? Learning how to code. Learning stats. Leaving the comfort of a spreadsheet’s visual display for R, maybe?" To make this statement would be ignoring that numerous non-free statistical packages have offered GUI front-ends for years. SAS/EG, JMP, Minitab, and SPSS instantly come to mind. Two out of those four are even marketed directly to people as statistical extensions of spreadsheets. Granted, the "long tail" will still need to learn a little bit of stats, but I fail to see how this is a problem that Excel solves (unless we're talking only about plots).

I don't think it's a hard leap to consider that if the big-box statistical package companies realized how much of their money came from industry, they'd do what they could to make their software seem like an alluring proposition. Statistical software costs an order of magnitude more than Excel, so they'd need pretty good arguments on how to sell upper management that the business team actually needed an 8000 dollar piece of software.

I'm not sold that there's nothing in between Excel and R. From my experience, they require a slight learning curve (nowhere near the learning curve of going from Excel to R), but not an insurmountable one. What these solutions lack is the name recognition that Excel has, or decent integration into a MS Office stack (exceptions, of course--I remember seeing a statistics toolbox for Excel once), or they cost too much.

I think part of the real problem is that for a lot of companies, Excel is "good enough". There's plenty of stuff it can't do well. It chokes on larger data sets, has limited statistical functionality, poor scripting capabilities, and shaky random number generators. But it's good enough for people who don't want to do much with their data.

If they wanted to do harder-core analysis, they'd outsource it to an analytics team. This perspective comes from a viewpoint inside BigCo. The prohibitive cost of some of these solutions might be a harder pill for a small firm to swallow.

Ph.D student in Stats here. That we can even sample the full posterior with so much data is interesting in and of itself. The method isn't particularly revolutionary, especially compared to a method Zaiyang Huang and Andy Gelman[1] came up with 8 years ago, but it's practical (for a restricted class of models). Steve Scott spoke to my department about a month ago, describing the problem. It's not a solution that works for all posteriors, but it certainly allows more freedom than restricting oneself to conjugate priors and allows computation on large datasets.

Academics tend to trivialize the implementation (we had some pretty strong critics of his talk in my dept), but some kudos are in order for that, even if the algorithm itself isn't revolutionary.

[1]: http://www.stat.columbia.edu/~gelman/research/unpublished/co...

If you're interested in experimental and noisy, you can't really beat Metal Machine Music. It's practically unlistenable ;).

We're interested in the probability of a coin flip yielding heads before we flip any coins. Uniform is just a really common prior to choose in this situation for a few reasons:

-It's a special case of the beta distribution, which is the conjugate prior for binomial problems. This means that the distribution of the probability of getting heads given the coin flips is in the same family as the prior itself (ie: beta priors with binomial likelihoods yield beta posteriors).

-The uniform (for this problem at least) is an "objective prior", which expresses that we don't have much information about whether the flip is biased. The example you give (modeling p^2 instead of p) is a great example of when the uniform would be a bad choice. The reason the uniform doesn't work in this case is because for binomial data (coin flips), a uniform prior is not invariant to reparametrization.

If choosing priors was so simple as always going with the uniform, there'd be little reason to go with Bayes! The choice of prior sometimes makes a radical difference in the posterior (especially with small samples), and there's many things to consider when you choose priors (computational convenience, uninformative versus informative priors, hierarchical modeling, etc).

http://en.wikipedia.org/wiki/Jeffreys_prior http://en.wikipedia.org/wiki/Beta_distribution http://en.wikipedia.org/wiki/Conjugate_prior

As lp251 says, the determinant method is not a good idea.

Checking the condition number of the covariance matrix of the LS estimates (X'X)^{-1} is the way R does it.

As someone who did graduate work in Bayesian methods for a statistics master's degree, I take offense to saying machine learning is not a concern of statistics and data mining (but not really)! The hesitance towards Bayesian methods seems more related to the discipline, and it seems that places that call what they do "machine learning" tend to be less hostile towards the explicit subjectivity of Bayes (I would highlight the word "explicit" in that sentence--Frequentism has it's fair share of subjectivity as well).

There was a great post on Stats.SE a few years ago about the difference between statistics and machine learning[1]. Leo Breiman once argued that statistics tends to focus more on model fitting and checking, while machine learning looked at prediction accuracy. The exchange between Andy Gelman and Brendan O'Connor is pretty funny. It has been my personal experience however that many people that apply a method that they brand as "machine learning" are not as bothered with assumptions as my fellow conservative statisticians.

But statistics and machine learning are quite similar in foundation. Barring the differences in terminology, as a professional statistician, I find I have as little difficulty read machine learning papers and algorithms as I do reading statistics ones.

http://stats.stackexchange.com/questions/6/the-two-cultures-...

I asked a similar question for R about a year ago, and saw an interesting way to do it, taking advantage of the math. http://stackoverflow.com/questions/11623865/faster-modulo-or...

It's not surprising, but still fascinating. This property isn't limited to humans and wolves and dolphins, either. Scientists at Cornell showed that parrotlets have unique "names" given by their parents. http://news.cornell.edu/stories/2011/07/parrots-learn-their-...

I remember my mom telling me stories about Sun's glory days, and me thinking that it seemed sort of unbelievable that any workplace would be as wild as she described. Interesting to see a lot of the same sort of stories being told here and more. She was there from 1988 to 2008. I wonder what the Solaris, mysql, and Java teams are like now that it's run by The Evil Empire.

I encourage you to peruse the Github page of the R celebrity Hadley Wickham. While trained as a statistician and employed as a professor of statistics at Rice, he writes quite a few R packages that deal with more traditional development roles (unit testing and debugging, not to mention Plyr and GGplot): https://github.com/hadley?tab=repositories

Also, Duncan Temple Lang, another professor of statistics, has done some really amazing work with R compilers and CUDA interfaces. https://github.com/duncantl?tab=repositories http://www.omegahat.org/

To your last point, it depends on what role you currently serve to your company. If you're among statisticians, you make yourself valuable by knowing more about computer science and programming than the rest of the group. If you're amongst engineers (probably more common than the former--at least on HN), knowing probability and statistics gives you that edge.

That's the one I was looking for. Thanks!

I, for one, am glad that IEEE is making an effort to get people excited about a formal education in data science. My education was a disjoint combination of CS and Statistics (my degree is formally in statistics), with no union between the two except what I made of it. In neither CS nor Statistics did my education formally cover problems associated with having too much data to fit in memory or store on one hard disk.

My biggest issue with the teams of statisticians I've worked with before is that they lack a basic understanding of computer science. My biggest complaint dealing with the software developers on analytics projects is they don't understand statistics. I heard a great quote for which I don't remember the source (I paraphrase here): "A data scientist is someone who knows more computer science than a statistician, and more statistics than a computer scientist." The nature of the analytics world right now suggests that this type of specialty is sorely needed in many places.

What makes computer science a "hard" engineering and statistics not a "hard" subject? I'd hesitate to call statistics an engineering discipline, granted, but I object to differentiating statistics from CS on the basis of the word "hard".

I hope you're not trivializing statistics simply because of the elementary approach most schools tend to take for the first three or four statistics classes offered at the university level. I would argue that statistics can be just as difficult as computer science, and just like CS has intractable problems, so too does statistics.

If your idea of big data is acturial tables, you're not thinking big enough. Actuarial tables are largely still small enough to be handled in orthodox ways using orthodox software (SAS has a deathgrip in managed health--what I do for a living--and it's a 50 year old piece of software). In fact, most of the times I hear people say their data has a very high volume, it really doesn't, and the traditional methods of data analysis can handle it. Once you step into Facebook/Amazon/Google sized data, things change. And that's only looking at "big data" as large volume--the most common definitions of the phrase involve several other variables (see: the four V's).

Just because what's commonly done or visible is elementary doesn't mean the entire field is elementary. It would be like assuming CS isn't a hard engineering if all you saw were web designers/developers.

Couple comments: Anyone in statistics/probability has seen this quote multiple times: "All models are wrong, but some are useful." -George Box.

We often don't know the underlying distribution of a particular process or our data, true, but that doesn't mean that we can't get useful information out of it by making some assumptions, provided that our assumptions are not wildly wrong. For example, if we assume height is normally distributed, that should mean it's possible, under our model, to see someone who is -9000 feet tall. We're willing to accept this falsity in exchange for having some nice properties we wouldn't have had otherwise, provided we exercise some common sense.

In regards to (3), if all you can wield is the SLLN, you can't do much inference or prediction. So you're able to guarantee that the sample mean will converge to the expectation almost surely. The closeness of approximation in a finite sample case using asymptotics is intrinsically tied to sample size and the particulars of the underlying distribution of the random variable.

In regards to (5), the curse of dimensionality does become a large problem, but throwing out assumptions makes it a far bigger problem. The nonparametrics community has a lot of trouble with multivariate distributions for precisely this reason! This is one area where parametric models do better than non-parametric ones, since they have so much more structure to play with, it makes the problems so much more tractable.

You mention order statistics... order statistics in higher dimensions is a tricky concept that is not as defined as the one dimensional case. Which one is more: (3, 2, 1) or (1, 2, 3)? Wouldn't that necessarily depend upon the underlying distribution or some other measure of distance? If you have a paper to suggest that general high-dimensional order statistics makes sense, throw it this way. You are correct that Least Squares doesn't require a distribution to get mean and variance estimates, but what good is that if you don't know how much of your population fits within a multiple of your standard deviation around your mean? You could use something that works for all distributions like Chebyshev's Inequality, but to get that level of sweeping generality seriously hurts your power (If your data IS actually normal, 95% of obs fall within 1.96 SD's of the mean. Chebyshev will tell you 75%), and even Chebyshev had to impose a restriction to get that bound--it assumes finite mean and variance.

While there are distribution-free hypothesis testing methods, most of classical nonparametric statistics makes some assumptions about the underlying distributions, albeit they are far gentler than the parametric assumptions.

Always choosing nonparametric methods is just as short-sighted as always using parametric methods. It's very possible that you are throwing away vast amounts of information by using a nonparametric procedure when a parametric one would have been completely adequate. With ANOVA and two-sample tests, you don't lose much by going with Kruskall-Wallis or Mann-Whitney tests (under normality, MW has a relative efficiency of 3/pi versus the t-test), but in other circumstances, using a nonparametric method could be way worse, provided the parametric assumptions are true.

What I'm getting at is that while non-parametrics is nice in that it doesn't require many assumptions, they may be throwing away too much. Fitting to a distribution may very useful in making inferences and predictions. All models are wrong, but some are useful.

It depends a lot on what you do for work, I bet. I'm in sort of a similar situation as you--Windows at work, Linux at home, but I also have a mac. I run a pretty bare-bones setup on my Linux desktop (#!: an OS for those who got frustrated trying to learn Arch and still like APT). The lack of extra stuff makes me less prone to messing around, so it's great as a GSD system and when I want to dedicate my CPU and memory to the primary task instead of to the overhead of a bulky GUI and other features. Having a CUDA-capable GPU is nice, too, instead of having to use an emulator. If you do technical or scientific computing, Linux may be a good bet. The vagueness of my advice is because I'm only involved in a small subset of the technical computing world, but a lot of the statistical programmers in the FOSS world use Macs or Linux boxes (Debian-derivatives are very popular in the R community).

OS X probably has a bigger advantage for front-end and mobile dev, though.

If your decision is based purely on aesthetics, given enough time, I'm sure you can find a GUI for Linux that you like more than OSX. While OSX gives you a GUI with no fuss, the customizability of a lot of desktop environments for Linux means you can tune it to exactly what you want, whether you want more graphical flare, or less (for the OpenBox folks out there--represent!).

Your first point is not just a property of the Gaussian, but of all distributions. It's the Second Axiom of Probability!

Your method is incorrect as a method of normalization, but it seems we are in agreement that perhaps normalization itself is a silly method for this. Most instructors I've known that claimed they were "curving" were in fact using the percentile approach. Grading based on percentiles (top 10% get A's, next 20% get B's, 40% C's, 10% D, bottom 20% F, for example) allows you far more flexibility, as the percentiles don't even need to be symmetric. My grad courses usually only had three grades: A+, A, and "Please see me to discuss your future in the department".

Uniform's pretty much the go to distribution for stuff like that--or having a non-random piece and a small random component. That's how Pokemon determined Catch Rates[1].

The problem with a lot of these distributions is that while you may have a closed form expressed for the PDF, it doesn't always give you an easy way to generate samples from that distribution. There's a huge amount of research that goes into efficient ways to generate samples from these distributions, and they are often quite complex once you are unable to use the inverse probability transform [2]. There's a reason that the Metropolis-Hastings Algorithm was such a big breakthrough, and that's because sampling is often quite difficult. In the multivariate setting, sometimes you may have to throw away 10,000 random numbers before you finally get one that could come from your target distribution.

For most of the univariate distributions, it's not tremendously difficult to sample, though, so you may be okay.

[1]: http://bulbapedia.bulbagarden.net/wiki/Catch_rate (this article is total nerdvana, by the way) [2]: http://en.wikipedia.org/wiki/Inverse_transform_sampling

To explain why is difficult in layman's terms except for a few cases (the binomial is the token example for classical CLT).

For the majority of the people I had in the intro statistics courses I TA'ed for, this was enough explanation. In undergrad, we didn't prove Lindeberg-Levy CLT until my third quarter of math stats, and the versions with weaker assumptions we didn't prove until graduate math stats.

When you're struggling to understand the fundamentals of hypothesis testing and simple combinatorics, going through the rigorous process of proving the CLT is daunting, mind-boggling, and largely unnecessary :).

Btilly outlined the steps of how you'd go about showing mathematically that a binomial converges to the normal distribution, but the reason you don't see this approach, even after two courses in statistics, is that it takes awhile for people to wrap their heads around the steps. At most universities, calculus (more than just a simple derivative and integral here and there, I mean) and real analysis isn't introduced to statistics until students take math stats, which is something you'd take after you were well-versed in the applications and shapes of several common distributions. The Bean Machine is usually a more intuitive way that students are introduced to "why" binomials become normal as the sample size gets larger.

The reason that _so many_ things have normal distributions is because of the Central Limit Theorem. If there's any reason to study math stats (other than understand the underpinnings of statistics at a deeper level than most of the population), it's that you get to become really familiar with how amazing this Theorem is.

I've surfed around on this article several times while trying to find a distribution I needed in order to make my rejection samplers work better for distributions that have no name or are obscure (posteriors in Bayesian inference often generate some totally whacky distributions--especially once you get out of the realm of using conjugate priors. Here, there be dragons!).

If you aren't reading probability from a textbook, this article isn't going to tell you which distributions are the most useful. The Beta Distribution receives half the amount of space as the Dirac delta function, but for you folks who want to take a Bayesian approach to A/B testing, the beta is going to be far more useful (The beta is the conjugate prior to the binomial) than the dirac, though one might argue that the Dirac is interesting from a mathematical perspective, it being a degenerate distribution and all...

It turns out though that many of the "useful" distributions (by which I mean the ones that many classical problems were devoted to) share very similar properties. If your aim is to get familiar with the most common distributions encountered in the wild, the article on the exponential family will serve you better than this overwhelming article.

http://en.wikipedia.org/wiki/Exponential_family