tristanz

Sense - https://senseplatform.com - San Francisco

We're building a next-generation data science platform to tackle the toughest problems in science and business. We're looking to expand our team of two. If you love tackling tough problems, building serious technology, or spending your days thinking about future of data and statistics, drop us a line.

We're hiring multiple roles:

* Fullstack Engineer - Build the best cloud platform for data science. Node.js/AngularJS.

* Backend Engineer - Build the best infrastructure for data science. Hadoop, Docker, AWS, Go, R, Python, JavaScript.

* Lead Designer - Build the most beautiful, powerful, and productive data science experience.

* Director of Data Science - Build the future of data science and spread the gospel. PhD and polyglot data science experience (R, Python, JavaScript, Hadoop, SQL) required. Bayesian/Probabilistic Programmer a plus.

Compensation: Competitive salary, real equity, flexible work environment.

Sound interesting? Email tristan@senseplatform.com with a short introduction.

The Cuban comment in that thread is interesting:

"I didn't miss a thing. When a company raises hundreds of millions of dollars I would have been diluted to nothing. But the bigger issue is that I'm not a fan of situations where you have to raise hundreds of millions of dollars to do tens of millions in sales. It's a lesson learned from the tech bubble

It was one thing when the valuation as a multiple of sales was in stock. It's a bigger thing when that multiple is my cash."

Putting aside bandits and all that, it seems like the first step should be to set up a hierarchical prior which performs shrinkage. Multiple comparisons and stopping issues are largely due to using frequentist tests rather than a simple probabilistic model and inference that conditions and the observed data.

Gelman et al, "Why we (usually) don't have to worry about multiple comparisons" http://arxiv.org/abs/0907.2478

This is also typically written in algorithms like:

  N = rows(data)
  for i in 1:N
    blah

So even though 1:0 may seem not unnatural, it actually reads perfectly in most code.

I'm guessing you don't translate algorithms in scientific papers. The reality is that having one notation for mathematics and programming is a convincing argument to many. More broadly, this indexing thing is trivial. Let's not pretend Dijkstra's paper is proof of anything. At the end of the day his argument is just style: "That is ugly, so for the upper bound we prefer < as in a) and d)"

http://www.cs.utexas.edu/~EWD/transcriptions/EWD08xx/EWD831....

Sense - https://senseplatform.com - SF, NYC, REMOTE

We're a just launched startup building a radically better platform for data science. We've received tremendous response from both individual data scientists and CTOs at Fortune 500 companies. We're a tiny team of two PhD developer/statisticians and are looking for #3. You should believe in our mission and want to join a startup as a full partner.

Position: Full-stack web lead. You will be responsible for leading our frontend and backend web efforts. You should have extremely strong JavaScript skills and experience building large client-side JavaScript applications. Interest in data and data science a huge plus but is not required.

We're located in San Francisco but are building a distributed team. We're looking for the right person over everything else. We are open to almost any arrangement, including all the way up to adding a cofounder. If you have world class skills and are looking for an exciting project to join, let's chat.

Interested? Email: tristan@senseplatform.com

It's not necessarily about pushing the boundaries of math. The simpler you can make the math the better for a given level of insight. But I don't think it's "easy" to derive clean mathematical results for many economic models. It's a combination of math, insight, and style.

Imagine this: prove to me some conditions under which equilibrium exist for prices in a market of consumers and producers with heterogenous preferences and skills, that can only communicate on a network with a cost per interaction. Now introduce time and savings and prove the equilibrium interest rate over time. Now prove the optimal tax rate on capital and labor over the agents lifecycle. Now prove that when skill and preferences are not observable to the government. Now introduce default and unobservable savings. Are there conditions under which the markets unravel when faced with exogenous shocks to productivity or beliefs. What is the optimal interest rate policy?

These are classic questions in economics in a more realistic market that captures heterogeneity, limited information, and locality or search costs.

A problem with simulation in economics is that there is always a feeling that that simulation results are inferior to what could have been done if the writer was better at math. Let's say you derive some insight on heterogeneous agents transacting on a network. This is likely quite hard to tackle without simulation, but most of your peers reading the paper will hope there is some cleaner way to get the insight of the model. The result is that it's unlikely that economists will consider your result a classic, which is ultimately what is required to have a big influence in the discipline.

Anand and I built everything, including the choosing the colors of buttons. That's early stage startup life.

Scaling up from 10, 100, to 1000 cores is fast (3s per engine in parallel). However, something like 1000 cores would currently require spinning up new instances (1-2 minutes) if deployed in the cloud.

The distributed filesystem is meant for easily sharing code and medium sized data across containers. In the cloud, it is best to to use S3 directly for large data and local disks for high IO tasks. For on premise deployments, there are more options.

I'd be interested to hear about your use case. Feel free to drop me a line at tristan@senseplatform.com

The Python engine is IPython underneath the hood. Any code or visualizations that work in IPython notebooks should work in Sense.

There is a difference though. In our experience, we've found that the notebook style development, with code inline, is awkward when doing serious analytics. It's harder to use version control, editors, etc. We have opted for the dual pane experience common in R and Matlab. The output however can be rich and interactive just like an IPython notebook and is always saved.

Yup. It's just the two of us. So far. If anybody wants to join: tristan@senseplatform.com.

Yes, we're fans of how sharing and collaboration works on GitHub. The goal is to make Sense the center of gravity for data scientists the way GitHub is for developers.

We're not trying to replicate GitHub's features. The core of Sense is a better way to work with data: the compute infrastructure, engines, and analytics workflow. Advanced users using git will likely use Github in addition to Sense.

Yes. I'll also point our that Anand (apatil) was a core developer for PyMC. We have big plans around Bayesian computation on Sense.

You currently need an invitation code to register. We're giving these out slowly to make sure everything works smoothly.

Right now Sense is best for ad-hoc interactive analysis and batch/scheduled jobs. You can run long-running services that expose something like a REST endpoint, but we have plans to make exposing services much easier, so I'd probably hold off until we have an "official" solution.

Sense cofounder here. We're just getting started, so feedback is welcome.

Sense supports R, Python, JavaScript and SQL out of the box, but is fully extensible to new languages and tools:

https://github.com/SensePlatform/sense-engine

We have Julia, Hive, and Spark engines in development.

I have no idea if 21% is correct, but on Amazon Chromebooks are 4 of the 5 top selling laptops. Something is going on.

http://www.amazon.com/Best-Sellers-Electronics-Laptop-Comput...

It's unlikely to be much different than the recent Berkeley benchmarks, since they all ran on AWS: https://amplab.cs.berkeley.edu/benchmark/

This looks like the technology behind Prior Knowledge (http://priorknowledge.com/), which is not surprising given that a lot of the same people are involved. Awesome that it's going open source.

It will be interesting to see how far the database analogy can be pushed. The key thing to realize is that BayesDB is based on a particular model, CrossCat (http://probcomp.csail.mit.edu/crosscat/). If your database table is an adjacency list that represents a graph, for instance, it's not going to perform very well compared to a more tailored model. On the other hand, a generic approach to high-dimensional imputation is very useful.

DARPA and others think probabilistic programming: http://probabilistic-programming.org/wiki/Home

You need to understand Bayesian statistics enough to understand this paper: http://www.stanford.edu/~ngoodman/papers/WSG-AIStats11.pdf

The overall GPA can still be used to rank, even at Harvard. The latin honors are fixed at top 5, 20, and 50 percent of graduating classes.

I don't really care about grade inflation overall because I don't think more competition for grades would improve Harvard as an educational institution.

I do think grading norms/curves may be important to standardize the distribution across majors. It's absolutely the case that some students avoid difficult courses. This is very unfortunate.

My experience was that it's relatively easy to stack rank Harvard undergraduates, but it often doesn't feel like it serves any educational purpose to do so.

In the class I taught of 12, there were 10 that seemed like they deserved an A or A-. They completed all the assignments and did their absolute best as far as I could tell. Some were definitely better than others (A vs. A-), but giving the hardworking but not genius students Bs just made them stressed out. They would come to office hours to ask how they could improve. The only true answer I could give was "there are 2-3 students in this class that are just incredibly creative and gifted writers, they get the As."

Maybe it's the role of a teacher to stack rank students, but it's no fun. I'd much prefer to try to get the best out of each student. The truly top students are going to have no problem distinguishing themselves later in life anyway, it's just so obvious how good they are.

This is an absolute work of art, if you take the time to digest what they've done. In order to articulate a model of cognition, they have implemented a probabilistic version of Scheme, that compiles to JavaScript, and that allows for forward and reverse simulation in the browser. Given a model that can simulate forward from causes to effects, you can simulate backward (perform inference) from observed effects to causes, all inline throughout the book.

All of it's definitely solvable, it's just harder than lots of Monte Carlo problems that are embarrassingly parallel.

This is the core calculation to all of Bayesian statistics, so it's used pretty much everywhere. Almost any time you try to compute p(unknown|data) you run into computational issues. It's a high dimensional integration problem that has no closed form solution. There are models that can be simplified, but they are the minority.

Most Monte Carlo methods used to calculate Bayesian posterior distributions are iterative, you can't parallelize each draw since it depends on the previous draw. Moreover, each draw depends on a function of all the data, so if your data is larger than can be stored on a single computer it gets complicated.

This paper doesn't look like it has a magic solution though. It seems to propose an adhoc sharding approach and then shows it sometimes works.

Astronomy and cosmology are not science?

The paper he cites is by David Romer, but it definitely does not say you should go for it 100% of the time.

http://elsa.berkeley.edu/~dromer/papers/JPE_April06.pdf

"Teams’ actual choices are dramatically more conservative than those recommended by the dynamic-programming analysis. On the 1,604 fourth downs in the sample for which the analysis implies that teams are on average better off kicking, they went for it only nine times. But on the 1,068 fourth downs for which the analysis implies that teams are on average better off going for it, they kicked 959 times."

I tend to agree with Ian that it's confusing to conflate probabilistic programming and libraries that support Bayesian inference. In PyMC variables represent random variables, but these variables can't be used with Python constructs like conditionals and loops. Python is used to construct a DAG, which is then executed.

I think a better definition of probabilistic programming languages are languages where you can replace any variable of type T with Random<T>. The line isn't entirely clear, but library approaches in languages like Python don't fit, since they can't handle control flow. Bugs/JAGS/Stan might qualify, although they are very limited declarative languages. Their motivation is primarily to have a compact modeling syntax, not a real programming language.

There's no need for probabilistic programming languages to be some esoteric DSL. You can convert languages like Python or Matlab into a probablistic programming language with a lightweight compiler transformation: http://www.mit.edu/~wingated/papers/lightweight_pp.pdf. Actually doing inference efficiently, however, remains as challenging as always.