PourOver: A library for simple, fast filtering and sorting in the browser | Hacker News Reader

PourOver: A library for simple, fast filtering and sorting in the browser

nytimes.github.io

224 pointsjsvine49 comments

So I visited one of the PourOver examples, this Academy Awards fashion feature published earlier this year:

http://www.nytimes.com/interactive/2014/02/02/fashion/red-ca...

I opened the dev tools to inspect the traffic and code, and this pops up in the console:

              0000000                         000        0000000
            111111111      11111111100          000      111111111
            00000        111111111111111111      00000      000000
            000        1111111111111111111111111100000         000
            000        1111       1111111111111111100          000
            000         11       0     1111111100              000
            000          1      00             1               000
            000               00      00       1               000
            000             000    00000       1               000
         00000            0000  00000000       1                00000
       11111            000 00    000000      000                 11111
         00000          0000      000000     00000              00000
            000        10000      000000      000              0000
            000        00000      000000       1               000
            000        000000     10000        1     0         000
            000        1000000 00              1    00         000
            000         1111111                1 0000          000
            000          1111111100           000000           000
            0000          111111111111111110000000            0000
            111111111        111111111111100000          111111111
              0000000              00000000              0000000
       
       
       NYTimes.com: All the code that's fit to printf()
       We're hiring: http://nytimes.com/careers

....You sneaky audience-targeting bastards

> NYTimes.com: All the code that's fit to printf()

This gave me a chuckle. I was at a hackathon that the NYTimes sponsored a few years ago, and my group used almost this exact joke for our project, an ncurses reader for browsing the New York Times in the terminal[0].

I don't know who thought to put this in the source code, but I'd find it very amusing if they got the inspiration from hidden slogan from a hackathon project. Kind of like the old competitions in the 50s/60s to create company jingles.

[0] https://github.com/ChimeraCoder/ncnyt

I'd almost forgotten we made that joke. It wouldn't surprise me if that's where they got it from. The NYTimes employees at the hackathon seemed to like it. At least enough to tweet about it.

Mozilla has been doing this for a while if you open a developer console on MDN.

This has been in the console for all NYT pages since the new-site launch (I want to say it was there during the pre-launch beta). I spent some time trying to determine if the 0's and 1's meant anything, but I came to the conclusion that they were chosen more for stylistic purposes than anything (see the vertical line of 1's).

If you must know, I was in the console while developing a tiny userscript to add a link to images to open the superJumbo version of the image in a new tab.

Half the comments so far are looking for a non-code demo, and I imagine they mean something visual, so here's a bare-bones visualized version of the Basic PourOver sample code:

http://gojs.net/temp/pourover.html

It just takes the queries and uses the resulting data to make some nodes in GoJS (Disclaimer: a Diagramming library I develop. Not free, but easy to set up with this kind of data and see stuff fast).

If I had more time I'd make it prettier. The results are just visual representations of the data results getting filtered. Its very easy to take PourOver's example collections and data-bind some stuff to them (color of the nodes is data-bound to monster gender, etc). I'm sure it can't be hard to do the same in other data-bound visualization libraries.

This is very cool. I'll try to make a much prettier example tonight.

Oh this is great! I really should port something like this to the docs.

Hi, all.

A lot of folks are asking for a demo and, you're right. I should have included one. My apologies. I'll get to working on one as soon as I can.

In the meantime, I encourage readers to check out the source for http://www.nytimes.com/interactive/2014/02/02/fashion/red-ca... That's probably the clearest "demo" of PourOver at the moment.

More to come!

Thanks for doing your 'open source gardening'¹ so exuberantly!

I didn't see any contact info on your profile to let you know that the proportion of folks here using Ghostery² or Disconnect³ is probably higher than normal. Installing one will reveal how things break when JavaScript libraries [usually orthogonal to page functionality] are blocked.

¹https://news.ycombinator.com/item?id=7593242

²https://www.ghostery.com/

³https://disconnect.me/

This page needs a giant "demo" button near the top. The examples are all code.

Reminds me of crossfilter (http://square.github.io/crossfilter/) by square. It has a killer demo, however.

Crossfilter was absolutely the inspiration. I just wanted to make a Crossfilter that worked with the arbitrary boolean composition, could be dynamically added to and updated, and supported some of the less numerical patterns I was encountering.

Coolies =).

Would you ever add some of the Crossfilter functionality, to make it easy for people to plug in their own data to PourOver?

I would absolutely like to add a Crossfilter Filter type. I'd even accept a pull request ... ;)

Noice! Btw, the docs page is acting funky on an iPad (iOS 7.1). The layout seems to be alternating between mobile and desktop with each pan/scroll event. [ps: only happens in landscape orientation. portrait is unaffected]

It seems similar to our Dynatable plugin [1], which is basically the functionality of this plugin with some additional table-read/write functions included. The main difference being that this library depends on underscore, while Dynatable depends on jQuery (which is mainly used for its browser compatibility functions).

Given both library's emphasis on speed, it looks like I have something to benchmark against!

[1] http://www.dynatable.com

This looks really cool. I wouldn't say PourOver is focuses on speed as much as focused on being fast enough to get 60fps for 100k items. Cheers!

I much prefer a basic construct that makes it simple to do filtering and sorting that can be simply extended to any complexity. Sorting and filtering are really nothing more than set manipulation (which they state themselves) so with simple data binding this becomes a trivial exercise to build an impressive client-side search.

In Ember that might look like this:

    Ember.ArrayController.extend({
        filterA: Ember.computed.filter('fieldName1', function comparator() {}),
        filterB: Ember.computed.filter('fieldName2', function comparator() {}),

        joined: Ember.computed.union('filterA', 'filterB'),
        
        filtered: Ember.computed.uniq('joined'),

        sorted: Ember.computed.sort(function comparator() {})
    });

This is exactly what PourOver offers. You can chain filter results to any boolean complexity, as well extend the default filter types to optimize indexing and caching. Indeed, PourOver was trivial to make, an outgrowth of the very pattern you define above. PourOver is just an attempt to scrap that boilerplate, allow for the queries to be indexed, combined with sorts, and automatically rebuild when the collection changes.

Thinking about it more, I believe that my visceral reaction was to the imperative nature of the library. I was imagining trying to code something that generated a filter (e.g. PourOver.makeExactFilter("mythology", ["greek","norse"]);) and it seems like it would be unnecessarily complex without data-binding propagation/invalidation.

If I wanted to add "roman" mythology to that filter I would imagine something like:

    var mythologies = ["greek", "norse"];
    // ... create a collection with filters
    mythologies.push("roman");
    PourOver.makeExactFilter("mythology", mythologies);

But you don't get that last statement for free, nor the one where you join it into a collection, nor do I see a way to replace the previous mythologies filter. Any time you need to reprocess a filter you've got to run it through a series of imperative actions triggered from one of the events, and possibly throw away the collection and regenerate it (if you can't remove disjoint filters).

I would be pushing for a Object.observe/dirty checking/get-set version of this to take PourOver to the next level of utility. (?/Angular/Ember)

I think I'm a little confused. Why wouldn't you have just added the "roman" possibility from the get go? Could you provide an example of a situation when you don't know the universe of possibilities in advance?

If the user adds one, or new data enters from an outside source (pub/sub, sockets, etc)

Alright, this is a great point. It would be trivial for me to add addPossibilities. I think I'll do that! Thanks. This will really help for using PourOver to power tagging selection fields where you can ... add possibilities!

Exactly what @rattray said. It's something you get nearly for free in data-binding world and is much harder to accomplish in imperative code. I wish you luck, and make sure you blog about it when it's done! (I want to see how it's implemented.)

More details at http://open.blogs.nytimes.com/2014/04/16/introducing-pourove...

There are a few links to projects at the nyt that has used these two libraries.

I wonder if the misspelling is intentional. (The idiom is "pore over".)

I'm sure it is - "pour over" is a way of making coffee, and its sister project is called "Tamper" which is also coffee related. The "pore over" double meaning is clever though.

Indeed, looking at the logo again, it appears to be a pour-over brewer - clever. (I am a coffee neophyte and wasn't aware of this brewing method.)

Off-topic, but related.

Does anyone know of any JS libraries that allow for Excel-like interaction? However I am looking for something that just implements the presentation layer.

The problem I am running into is that the libraries that do exist conflate the presentation and data layers and try to do everything for you. In other words, they make the assumption that all data will be downloaded to the client and then manipulated on the client. These libraries do not fit my use case.

I am actually in the middle of an implementation of a grid view myself; I'm giving Handsontable[1] a whirl and going to look at using pourover for filtering my dataset. Definitely meets your Excel-like interaction though not certain that it doesn't try to do everything for you since I haven't made it that far yet. (Looks promising at least.)

1. http://handsontable.com

If you going to handle that many obejcts in the browser wouldn't you use something like pouchdb? http://pouchdb.com/

Also "Pagination strategies with PouchDB" http://pouchdb.com/2014/04/14/pagination-strategies-with-pou...

And here I thought I would be reading about the App.net PourOver posting service[0], which the NY Times Opinion account uses for posting[1]

[0]: https://directory.app.net/app/255/pourover/ [1]: https://alpha.app.net/nytopinion

I've used the https://mixitup.kunkalabs.com/ library for similar functionality in the past. I am not sure how well it works with hundreds of thousands of items, but it does work well for the 600ish items that I am using it for.

Is this basically what Google Refine[0] does with their facet filters? (but obviously open source and allowing anyone to build their own platform)

[0] - http://openrefine.org/

It would be a great move to isolate the Views/UI part of this out from the rest of the code base, which would make it more usable inside other MV* frameworks like Angular or Ember.

Benchmarks comparing PourOver to Backbone would be nice. Anyone?

There are some benchmarks against the base underscore filter methods in the tests. These are what Backbone uses by default. However, the real utility of PourOver is less in its raw speed -- I'm sure that it could be improved by smarter folks than I -- but in the patterns it abstracts. I hope you find it useful.

Having been recently exposed to the benefits (specially performance-wise) of lodash vs underscore, I wonder why it depends on the later.

From a glance of the source it doesn't look like PourOver is doing any heavy lifting with Underscore. PourOver works lodash.underscore build of Lo-Dash without issue though because it's not doing any heavy lifting you probably won't see a perf difference.

This is awesome, and exactly the type of thing I've been looking for. Thanks so much for sharing! Can't wait to try this out.

I love how New York Times has the coolest devs. Anyone here who works there? What's it like?

I do. It's pretty fun.

This might be a bit embarrassing ... but a couple of years ago, Dan Sinker came by and asked a few of the developers in the newsroom to talk about exactly that question.

Here's me: https://www.youtube.com/watch?v=k3HanxEFFlk

Here's David Nolen (although he won't be around for much longer ;) https://www.youtube.com/watch?v=qOdoCqoqXlU

Here's Tiff Fehr: https://www.youtube.com/watch?v=8GOWLNWifxc

nice...but yikes that seems like an intimidating place to apply for a job...you guys are all heavy hitters...

I totally wouldn't be intimidated. We all just love what we do. The best way to get a job is often to get to know the folks on the team, hack with them on projects. I'd really hate to work at a place that I felt was exclusive.

Oh I didn't mean to imply it was exclusive, just that I'd feel like I was out of my league. Like I've built a few things with D3 but I still find myself getting confused sometimes, and I've gone through some of David Nolen's clojurescript posts / tutorials and I'm still not 100% clear on the advantages of core.async...but then again I don't use them all that often either.

Well, if you ever wanna chat about core.async or PourOver or jobs or life, I'm more than happy to do so: erik.hinton@nytimes.com Nolen's also a great guy and really serious about helping folks out with stuff like core.async. I'm sure he wouldn't mind jumping in an IRC chat to guide you along!

I interviewed there almost a year ago for an iOS position, and it was an awesome atmosphere. There was a certain energy to the place and everyone knew a ton of stuff in the domain. I didn't get the job, but had fun anyways for that day in NYC.

Anyone have a demo?

Similar to crossfilter?

Reply on news.ycombinator.com