lil_tee

Thanks. The 13F XML is not too complicated, here is the relevant section of code in case helpful: https://github.com/toddwschneider/sec-13f-filings/blob/main/...

I haven't worked with XBRL though, so am not sure if it's more involved

GitHub repo available here: https://github.com/toddwschneider/sec-13f-filings

nytimes.com is #3 domain of late, really #2 behind only github.com if you exclude ycombinator.com (all of the Shows, Asks, etc.)

Top 10 domains by # of items on front page since 1/1/18:

   rank |     domain      | count 
  ------+-----------------+-------
      1 | github.com      |  2041
      2 | ycombinator.com |  1911
      3 | nytimes.com     |  1818
      4 | bloomberg.com   |  1028
      5 | medium.com      |   826
      6 | techcrunch.com  |   735
      7 | theguardian.com |   666
      8 | github.io       |   615
      9 | bbc.com         |   558
     10 | arstechnica.com |   493

FWIW, nytimes.com accounts for ~4% of all posts that appear on the HN homepage: https://toddwschneider.com/dashboards/hacker-news-trends/?q=...

Thanks, agreed this is a good idea and I added query param support here: https://github.com/toddwschneider/stocks/pull/4

TLC just released 6 additional months of trip-level data (Jul-Dec 2015): http://www.nyc.gov/html/tlc/html/about/trip_record_data.shtm...

GitHub repo updated to process additional data: https://github.com/toddwschneider/nyc-taxi-data

Number of bikes in active use does not seem to change with the seasons: https://github.com/toddwschneider/nyc-citibike-data/blob/mas...

The summer 2015 increase corresponds to the permanent expansion in August 2015; I'm not sure why there was a dip in Q1 2015

Yes, the database is all on the machine's local 512 GB hard drive. I did store the downloaded flat text files to an external drive and loaded them into the db from there.

Sure, there are undoubtedly lots of examples of businesses that opened in desolate areas and created new taxi activity where there had been none previously, I just happened to focus on Williamsburg for the post.

Another idea that I didn't get around to doing was to look at concert venues and measure taxi traffic around particular concerts to see if it would correlate to bands' overall popularity

Yep, all on a 2012 MacBook Air. Data size was over 400GB with indexes

Simple queries on indexed columns of the trips table take a minute or two, more complicated queries that require a full sequence scan can take up to a few hours

My dataset does not include livery cabs

FiveThirtyEight has some additional for-hire vehicle data in their GitHub which they obtained via FOIL request: https://github.com/fivethirtyeight/uber-tlc-foil-response/tr...

Intuitively, I would think livery cabs have lost significant market share to Uber, but I don't actually know

I did, though I asked permission from the Commissioner first

Mostly to keep the app simple and still deployable on the free Heroku plan. Sure, it could handle multiple locations, but then it would quickly outgrow the free tier

I think the projection becomes relevant when you're looking at "straight" borders like Colorado. A great circle will bend slightly north of the northern border, so you can actually just barely touch Wyoming and Nebraska, but I don't think the projection is significant when looking at some of the more irregularly shaped states

We have updated our post to incorporate some popular suggestions and reactions into our simulations: http://rapgenius.com/1504221

Simulation author here with some additional analysis using a faster distribution of request times. If you use a distribution with median 50 ms, 90th percentile 225 ms, and 99.9th percentile 898 ms, then you need 30 intelligent dynos to handle 9000 requests/minute without queueing. In the same scenario with 30 naive dynos, 44% of requests get queued.

Animations and results are in the explanation at http://rapgenius.com/1502046

I'm not familiar with minimum-of-two-random routing, but it does seem like assigning request to dynos in sequence would perform much better than assigning randomly (ie in a scenario with n dyno capacity, request 1 => dyno 1, request 2 => dyno 2, ... request n => dyno n, request n+1 => dyno 1, ..., repeat)

That'd be probably significantly better than the case of (request i => dyno picked out of hat) for all i

But you never want to have a queue on any of your dynos! A queued request means that a user is waiting with no response. If your goal is to have 0 (or less than epsilon) requests queued, it takes far fewer dynos if the requests are routed intelligently

If you have 10 dynos and 1000 simultaneous requests, the difference between naive and intelligent might well be reduced, but that's also a scenario in which your end user response times would be horrendously slow and so you'd need more dynos either way