hey Zjonsson, I think this is awesome, have you considered having a webworker run in the background to send the data to the browser and then switch once it has finished? The time to interactivity (and visibility) of crossfilter is a problem, and I think you did a great job solving that problem, I just don't find the interactivity to be as great as I was expecting (then again I'm on a slow connection currently).
A great idea, however the web workers would have to be semi-persistent as the initial build of the crossfilter object takes some time (i.e. I wouldn't spawn a new worker on a new request). Here I simply increased the heroku nodes to respond smoothly to Hackernews traffic. I'm pretty happy with the results.
Cool app! Besides crossfilter, is the source available somewhere? Thanks.
This is neat. Digging through the source to see how you're doing marker loading and clustering. One of the things I've been chewing on recently is making maps with many (>1000) markers behave smoothly. Seems like you've nailed it.
D3 is an excellent tool for managing a large array of SVG objects. Connecting it to leaflet is easy, just call function _initPathRoot() on the leaflet map object, followed by d3.select("#map").select("svg"). Then you can use map.latLngToLayerPoint(..) and map.latLngToLayerPoint(...) leaflet functions to calculate the X,Y coordinates in SVG space.