tristanz

SAS is the huge incumbent, but open source tools like R and Spark and cloud-centric platforms are going to win, no question. We're already well into this transition and tools like Sense should accelerate it.

Sorry about that! We just got a huge spike in traffic from HN and VentureBeat so are trying to handle the demand.

Hey all.

We just opened up Sense to the public. Happy to answer any questions you may have about Sense and building a modern data science platform.

Tristan

Sense - Modern Platform for Data Science - San Francisco -- https://sense.io

We're building a modern platform for data science -- a platform to help find new cures for diseases, build intelligent systems, and accelerate the pace of science and business.

We're looking for an engineer to join our core team. You'll be able to shape our entire product. You should have experience building products user love and be comfortable working on all parts of a modern stack, from a complex web frontend to distributed backend.

This is a role for somebody extremely smart and creative to make a massive impact on both a product and an industry. You'll be joining a group of like minded individuals. Every team member currently has a PhD or MS and are fully dedicated to bring data science to every organization.

Sound interesting? Email: tristan@sense.io

Simple examples looks fine, but try a setup where you have dozens of environmental variables being set, data volume-containers that need to match the lifecycle of the main container, pulling images that don't exist, ambassadors containers for cross-host linking, and other sidekicks for things like load-balancer registration and health checking. If you look at how Deis does it, they've actually created tons of little binaries to help with different tasks (https://github.com/deis/deis/issues/2254) because the unit files get too crazy by themselves. You end up trying to build a system like Kubernetes using shell scripts and unit files.

The key thing to remember with CoreOS is it puts systemd at the center not docker and only interacts with docker through the client. In fact, there's really zero docker integration besides shipping the binary.

There's also the underlying technical issue around what is PID 1. In your example, you aren't actually monitoring the container, you're monitoring the status of the docker client, which may be different than the docker daemon. For containers, the docker daemon owns the fork rather than systemd. In a system like Kubernetes you have a reconciliation loop that monitors the daemon. You could probably build a similar idea into your unit files to make them more robust, but that's even more complicated.

Having gone down the systemd route it seems like a dead end to me for distributed docker applications. If you believe in a containerized future, then choose an orchestration framework that treats docker as first order primitive. You can still bootstrap this framework using CoreOS/systemd but after that I'd much prefer to just interact with a higher-level system.

In my experience it works, but is a mess. You end up with really crazy unit files that aren't actually managing the docker daemon, they're managing the docker client. CoreOS created Rocket in part because the match between docker and systemd is so bad. Rancher seems to be going in the other direction by making Docker support fully native. I do wonder how much this really buys you over a higher level framework like Mesos or Kubernetes though.

You can see Darren's discussion of these issues here: https://groups.google.com/forum/#!msg/coreos-dev/wf7G6rA7Bf4...

This architecture seems great for analytics apps, but when expanded beyond that (e.g., Pete Hunt's Full Stack Flux talk) I never see explanations of basic patterns like validation. Where does validation happen and how are errors sent back to users?

For instance:

AddToCart(prod=1, quantity=1) -> Transactionally check that there is still inventory, return error if there isn't, add to stream if there is.

How do you handle disconnects and missed messages? What's required to make this a robust sync solution, including syncing the first query and then changes, akin to Firebase? Are there plans to support direct access from web clients and optimistic writes on the client?

Almost all the evidence is based on lotteries and addresses attrition. Expelling bad students is a potential problem, but students are tracked across schools. Some students are lost, of course. However, the evidence in the KIPP study shows that attrition actually raises the scores of the lottery losers more than lottery winners. This is because winning the lottery encourages some bad students to stick with it.

Most of the lottery evidence suggests that the gains from charter schools are largest for students that that are more disadvantaged (limited english, learning disabilities, and low initial test scores). So while these studies only give causal estimates for students that apply to the lottery, it's quite plausible that average effects would be even larger if all students were included.

Source: http://users.nber.org/~dynarski/kipp_may05_2010_web.pdf

I completely agree that this model is waiting for a fullstack framework to come along. Full fledged React apps are way too hard to get running on the server, even if theoretically possible.

A second component though is caching. Most fast server-side apps cache large portions of the rendered page. A built in cache layer could make things much easier. The test case is fully static websites.

Are there any actual examples of full-fledged isomorphic React applications? All the examples I've seen are toys. I mean isomorphic routing, flux architecture, async data access, etc.

Sense - San Francisco - https://sense.io

Fullstack/Frontend Engineer - We're looking for fullstack/frontend developers to help build a next-generation platform for data science and big data analytics built by and for data scientists. You'd be a core member of our team with the opportunity to make a massive impact.

We value the ability to learn quickly above anything, but you should have experience building complex JavaScript web applications (AngularJS) backed by developer-friendly REST APIs. Experience with data science tools and Hadoop/Spark a plus.

We're a small team, venture backed, and located in San Francisco.

Compensation: Competitive salary and equity.

Email tristan@sense.io.

Sense - San Francisco or Remote | https://sense.io

We're building a next-generation platform for data science and big data analytics built by and for data scientists. We're a small team, venture backed, and located in San Francisco.

JavaScript Engineer - We're looking for somebody to help accelerate development across our entire stack on a freelance basis. You should have experience building complex JavaScript web applications and developer-friendly REST APIs. Our stack consists of AngularJS, NodeJS, Postgres, Docker, Firebase, and lots of data science tools. Strong, demonstrable JavaScript/AngularJS/NodeJS skills a must.

Compensation: Competitive. We're located in San Francisco (SOMA) but have a office culture that supports remote team members.

Interested? Email tristan@sense.io with portfolio or resume that demonstrates NodeJS/AngularJS experience.

Sense - San Francisco or Remote | https://sense.io

We're building a next-generation platform for data science and big data analytics built by and for data scientists. We're a small team, venture backed, and located in San Francisco.

Fullstack Engineer - We're looking for somebody to help accelerate development across our entire stack. You should have experience building complex JavaScript web applications and developer-friendly REST APIs. You'd be a core member of our team with the opportunity to make a massive impact. Our stack consists of AngularJS, NodeJS, Postgres, Docker, Firebase, and lots of data science tools. Strong JavaScript/AngularJS/NodeJS skills a must. Experience in data science and Hadoop/Spark a plus but not required.

Compensation: Significant equity and competitive salary. We're located in San Francisco (SOMA) but have a office culture that supports remote team members.

Want to chat? Email tristan@sense.io.

Somebody really needs to write an open source Firebase clone. There's a gaping void in the market for something that doesn't lock users in and supports on-premise installs. This could be a pure open source project or open source business like MongoDB.

It doesn't need to be exactly Firebase. But Firebase sets a high bar. To my knowledge, there is no production-grade datastore that supports realtime sync, range queries, and eventual consistency in the face of intermittent connections.

Email tristan@sense.io if interested trial customer / sponsor.

Sense - San Francisco (SOMA/Remote) - https://senseplatform.com

We're building a next-generation cloud platform for data science. If you love data, statistics, machine learning, and challenging problems, you'll love what we're doing. We're a small, highly-technical team of 4 looking to expand to 5-6 in the next few months.

* Frontend Engineer - We're looking for a frontend-focused engineer who cares deeply about user experience. You should have experience building complex JavaScript applications. Our stack consists of AngularJS, Node.js, Postgres, Couchbase, and Docker, plus a whole bunch of data science magic.

* Data Scientist - Do you want to push the frontier of what's possible with data? We're looking for a polyglot data scientist to help make Sense the best platform for data science. If you love playing with R, Python, Spark, Julia, JavaScript, D3, and more, this job is for you.

Competitive compensation, benefits, and a flexible work environment.

Sound interesting? Email tristan@senseplatform.com.

Sense - Next-Generation Data Science Platform - San Francisco, CA

https://senseplatform.com

jobs@senseplatform.com

-------------------------

Frontend/Fullstack Developer

We're looking for a frontend/fullstack engineer to help build a radically better platform for data science. You should have experience building complex JavaScript applications and designing clean REST APIs. Our technology stack includes AngularJS, Node.js, Go, Postres, Firebase, in addition to a large data science infrastructure built on Docker. Interest in machine learningand data science a plus but not required.

We're a small team (4), venture backed, located in San Francisco, and offer competitive salary, real equity, smart colleagues, and work flexibility. Sound interesting? Shoot us an email introduction.

Email: jobs@senseplatform.com

Everybody brings their own terminology to these issues, given their background. This is all just statistics. While I think the intuition of your terminology makes some sense, it's probably the wrong frame. It's important for people to use same terminology, with a clear mathematical definition. Unfortunately there's still a big gap between statistics and machine learning communities.

Everything in "learning" follows a good parameterization of p(y, x), the joint distribution of the data, whether unobserved or not. If you have that, you can get everything else.

The core idea of representational learning is important, even though it's obvious in hindsight. When doing statistics, many people assume the parameters governing the conditional distribution p(y | x) are distinct from those governing p(x). So if you're just interested in predicting y, you don't need to model p(x) to get the posterior distribution p(y | x). Representational learning suggests that the parameters of these distributions are coupled. It's saying that if you understand the structure of the world -- you have a good representation of p(x) -- you can make better predictions p(y | x) with less data. This makes a lot of sense because the distinction between x and y is totally arbitrary.

Supporting docker caching would be huge.

It appears that AmisaDB is in-memory only, only supports read committed isolation, and has limited transactions semantics (no arbitrary JS). This is pretty far from what DocumentDB seems to offer.

http://www.amisalabs.com/AmisaDB_Docs.html

Sense - San Francisco, Remote - https://senseplatform.com

Sense is next-generation platform for data science and big data analytics. We're building something awesome, are venture backed, and located in San Francisco. We're a small, highly-technical team of 3 looking to expand to 5-6 in the next few months. We're open to remote for the right candidate.

* Frontend Engineer - We're looking for a frontend / fullstack engineer to help lead development of our web platform. You should care deeply building great products and have experience building complex JavaScript applications and designing developer-friendly REST APIs. Our stack consists of AngularJS, Node.js, Postgres, Couchbase, and Docker, plus a whole bunch of data science magic.

* Data Scientist - Do you want to push the frontier of data science rather than work on better ad targeting? We're looking for a polyglot data scientist to help make Sense the best platform for data science. If you love playing with R, Python, Spark, Julia, JavaScript, D3, and more, this job is for you.

Compensation: $90k-$120k, 1%-4% equity.

Interested? Email tristan@senseplatform.com.

What's the trend around external vs. eventual consistency at Facebook? I ask this comparing eventual consistency in TOA and Google's shift towards external consistency in Spanner.

Most probabilistic programming languages give you samples from the distribution of execution traces that are consistent with the observed data. No probabilistic programming languages actually run things backward. They all use standard Bayesian computational methods, like MCMC over possible execution traces. The beauty is the user can forget about this and just build generative models with primitives every language contains -- random numbers. The language then offers a way for users to query plausible values for the unobserved elements of the model.

Conceptually, the easiest way to understand things is via rejection sampling. You can "run things backward" but running things forward and just keeping the traces that match the observed output. That's Bayesian inference. Unfortunately, the probability of matching the observed output is zero for most problems so you need to use more advanced methods. Most accounts gloss over these methods since it's a huge research area.

Running things backwards is also not a perfect analogy because if fails to account for the probability distribution you start with.

There is a good tutorial here: http://code.cogbits.com/wiki/doku.php?id=start

It would be great if Cap'n Proto had better library implementations. This is one area where Protobuf, MsgPack, and, of course, JSON win.

Sense - https://senseplatform.com - Lead UX/UI Designer

At Sense, we're building next-generation data science platform to power the future of science and business. We're a small team, ventured backed, and located in San Francisco.

Designer #1: Lead Product/UX/UI Designer

We're looking for a talented designer to lead product, UX, and UI design at Sense. You'd be part of first 5 employees and would be responsible for crafting the entire product experience. You should love solving tough UI challenges elegantly, building beautiful and highly-functional user experiences, and working with creative and talented peers.

Competitive compensation: $80k-$110k, 1%-3% equity.

If you think you might want to join us, email tristan@senseplatform.com with a link to your past work.

Is there any story for persistent data / running database services?

Sense - https://senseplatform.com - San Francisco

We're building the most powerful platform for data scientists tackling the toughest problems in science and business. We're a tiny team (currently 100% PhDs), venture backed, and located in San Francisco.

We're making our first key hires:

* Platform Engineer - Go/Node.js/Docker/CoreOS/Hadoop

* Frontend Engineer - JavaScript/HTML/CSS/AngularJS

* Senior Data Scientist - R, Python, Hadoop, Spark, Probabilistic Programming

This is a chance join a startup tackling interesting problems in an area ripe for massive innovation. We offer competitive salary, significant equity, and a flexible work environment.

Sound interesting? Email tristan@senseplatform.com.