phunge

Altair is amazing. I've been using matplotlib going on 12 years now -- Altair didn't feel more productive at first (more typing), but the longer you work with it, the more you realize how the API is a cohesive system. This leads to a more iterative workflow and eventually to better visualizations.

A hazard rate can be thought of as a patient's one day probability of mortality. A hazard ratio is a ratio of two hazard rates, so 1.25 would mean that people in one group were dying at a 1.25x rate relative to a comparison group.

For Bayesian stats, "Statistical Rethinking" by McElreath is a masterpiece.

Neat! See also: https://fklearn.readthedocs.io/en/latest/, which wraps sklearn in a much-improved functional API.

Have a look at duckdb, it's is another interesting tool that's columnar and embedded.

It would be amazing if the world of open-source column stores matured a little bit relative to where we are today...

+1 to this, I've had multiple teams that have shied away from airflow because of its operability & deployment story, despite needing something that has all its features.

In terms of radically different takes on workflow engines, I'm very interested in reflow. I haven't used it enough to know if the rough edges are a deal breaker.

They're also used in spectral clustering, which is a powerful method for clustering data (see Ng 2001 for some pictures)

I think it's pretty easy for this discussion to become about personalities and not methodologies. Gelman is pretty brash and speaks his mind, so he could be correct on methodology and have that get lost in the noise.

So I'll give my opinion as a datascientist: null hypothesis significance testing is broken, the whole thing needs to get chucked. It's not fixable, and it's not only p-hacking that's the problem. Go read Frank Harrell, I like his writings on the topic more than Gelman's.

In cases where you need to interpret the resulting model, I've been advised not to bin (for example: http://biostat.mc.vanderbilt.edu/wiki/Main/CatContinuous). Other alternatives are splines or generalized additive models.

Classic post! This post is like a gentle gateway to the world of Bayesian statistics -- check out Cameron Davidson Pilon's free book if you want to go deeper.

Highly recommended! The first course was the first thing I came across that helped me contextualize the DL field into something that might be relevant for my work. It's a great way to get your hands dirty.

One point of comparison is Cam Davidson Pilon's Bayesian Methods for Hackers, they have a similar vibe: practical applied advice from a field that tilts towards the academic...

Logistic regression can easily be made online too, keep in mind! sklearn has an implementation of online gradient descent, and vowpal wabbit is also excellent at those problems.

Naive bayes can be parallelized in ways that SGD can't, that's a whole other conversation.

If you're interested in getting parse trees from regexes (ie all the submatch locations), check out Kleenex, it looks really interesting and is based on some new research. Full parse trees are a harder problem than matching, and one that is not a primary focus in RE2 performance AFAICS.

I'm super excited for this! Good dashboarding for python has been a huge gap in the ecosystem, and none of the existing options (bowtie/pyxley/spyre) really came close to the functionality of shiny.

The closest IME has been ipywidgets+jupyter and the jupyter dashboards server. Jupyter is really nice for developing, but deploying is another question. The dashboards server works but development on it has been stalled, and deploying it is a pain. Also, if dash can leverage the react ecosystem, that could make it pretty compelling compared to ipywidgets.

Agreed, this is a great way to think about it. I think Ansible is at its best when the modules handle the imperative stuff behind the scenes, and the playbook just declares. That's not always possible, but the modules are getting better and better.

FWIW I also found YAML to be very confusing syntactically at first; things got easier once I realized that it's basically 1-to-1 with JSON, and could convert to JSON to get intuition for the file structure (simple yaml2json script here: http://pastebin.com/TpjZLnLa). Thumbing through the book Ansible Up and Running also helped.

In the long run, putting a declarative idempotent layer atop the same old mutable infra is tough but a necessary compromise right now. It'll be great if the immutable-first tools of today (nixos et al) mature and we can leave this behind.

I actually like Travis quite a lot except for one issue! At work we use docker for builds, and neither tool can build docker images in such a way that the docker layer cache is utilized. For more details: https://github.com/travis-ci/travis-ci/issues/5358, https://circleci.com/docs/docker/#caching-docker-layers, https://github.com/docker/docker/issues/20316.

This makes our CI builds painfully slow. It's not entirely CircleCI or Travis's fault, it's the interaction between them and docker.

Which are the best of the SaaS options? Just curious -- I'm not happy with CircleCI or Travis right now.

Ooh Python! Check out https://github.com/mrocklin/multipledispatch which is a little more fleshed out than Guido's implementation. From the author of toolz and dask so you know it's good ;)

I also find that multiple dispatch is something that's occasionally handy, but single dispatch is truly everywhere. I think if you broke down multipledispatch uses in languages which embrace it (e.g. CLOS), the mix is like 95% single, 5% >single. For single dispatch in python, check out https://docs.python.org/3/library/functools.html#functools.s... which was added in to the Python 3.4 stdlib. The alternative is the OOP subclassing style, which works but IMHO isn't very readable.

They mention that it was originally paired with Druid. The data volume that Druid excels at (and that AirBnB must have) is orders of magnitude larger than what Tableau&Looker do well with. It's probably just built for bigger-than-SQL OLAP usecases.

Yeah this is a common complaint. I don't have direct experience with NaCL/libsodium or Keyczar but check them out. Do you mean a bitstream-compatible thing, or a GPG/PGP alternative?

I found that learning ansible and storing my machine customization as a deployable playbook was totally worth the effort involved.

To put up something concrete, my last large employer had a 401k plan where the cheapest fund we had access to (PLFMX) had an expense ratio of 0.72%. That's in addition to account-level management fees, which I think were ~0.8%.

This is 100% true. Mainstream 401k fees are out of control. I don't know how much is opportunism and how much is regulatory burden, but parity to wealthfront or anything else retail would be fucking miraculous.

This is an incredible resource, +1. I started using ledger heavily about a year ago and it's been a game changer.

Personal accounting is about data capture; if you don't gather good data, you end up with garbage in garbage out analysis. Having a simple, non-propretary file format enables so many things. The data capture side (OFX/QIF/CSV files, bank websites, etc. etc.) is still clunky and painful but that's a hard problem to solve.

We had a similar desire: secure, performant egress from S3. We ended up using CloudFront. You can configure cloudfront with signed URLs, that will access private buckets with a special S3 user (http://docs.aws.amazon.com/AmazonCloudFront/latest/Developer...). Any app that holds the private key can sign URLs to download from CloudFront.

It works great, but it really bugs me that we had to do that. The default download speeds from our buckets on S3 are atrocious. We store big datafiles in S3, and our development flow involves downloading them lot. If Amazon had an upgrade to S3 so downloads by chosen users weren't throttled or slow, we'd pay for it in a heartbeat.

You're probably thinking of https://en.m.wikipedia.org/wiki/Selective_laser_sintering which is pretty widespread -- for example you can order metal parts straight off ShapeWays that are SLS!

Can someone explain why the random_split (i.e. first) rule tends towards inequality? It seems that any time a rich&poor person meet, the expected outcome is that the poor person receives money -- E(Uniform(0, X+Y)) = (X+Y)/2 > min(X,Y). So how does that lead to a rich-get-richer effect?

The author wrote Luigi at Spotify, which is a pretty good example of a small amount of code with immense leverage on other people's productivity.

Preach! :)

That's my preferred design too. The first job of any external data capture process is to capture the full fidelity source data. Everything else belongs in a followon job.

Man, if I had a dollar for every time data roundtripping to files has corrupted things I'd have many dollars.

The schema-based serialization libraries (Thrift, Protobufs) or MsgPack are a good way to avoid that too. They come with a lot less baggage than say HDF5. Also, if efficiency isn't paramount -- just use SQLite! Amazing tool when it's in its sweetspot.

Lots of tradeoffs when dealing w/ serialization and file formats, no easy answers.