Creating Heatmap from Scratch in Python (2018) | Hacker News Reader

Creating Heatmap from Scratch in Python (2018)

geodose.com

70 pointsgeomatics9923 comments

I like seaborn's jointplot()[1] function for generating plots like this. I actually prefer the "hexagonal bin" plot, but jointplot() can also do kernel density estimation like this post does.

Hexbin example: https://seaborn.pydata.org/examples/hexbin_marginals.html

KDE example: https://seaborn.pydata.org/examples/joint_kde.html

If you are doing any exploratory analysis using scatterplots, I highly recommend that you also generate hexbin plot or heatmap. Far too often it turns out all of the points are concentrated in a small area of the plot, but that's not always apparent on a scatterplot.

[1] https://seaborn.pydata.org/generated/seaborn.jointplot.html

Please be aware that you MUST NOT use this for geographic coordinates such as "GPS" or "WGS84" latitude & longitude pairs. Depending on the longitude areas are of different sizes. You must use code that knows about that or your results will be utterly wrong.

The OP makes a related mistake (although for their data, it worked out OK). It uses this grid:

  x_grid=np.arange(x_min-h,x_max+h,grid_size)
  y_grid=np.arange(y_min-h,y_max+h,grid_size)

where h is a small constant, and the range ends are the min/max of the data. In general, this will result in rectangular pixels, which you probably don't want.

In your example, even for a small geographic area, a degree of longitude will typically translate to a smaller distance than a degree of latitude, so as you say you can't treat them the same.

This would only skew density maps not counts maps.

Well, this is about density maps unless I overlooked something?

How does this factor for tools like Datashader?

Same idea for datashader.

If you're overlaying datashader on a map, use the projected coordinates as input (to both datashader and/or any heatmap).

I much prefer kriging, which seems to better model sparse areas. https://en.wikipedia.org/wiki/Kriging too bad there are no image comparisons for those methods. Main difference - instead of sum of distances with cutoff, you make sum of values weighted by distances for each map point.

What you describe sounds more like idw than kriging.

Indeed, I've started with nice images on web which were described as "kriging", then went on to page about kriging, seen some general ideas and implemented what I thought would be fast, simple and work. That way I had something which looked right for my application (visualising district heating temperatures in city), but only now I know how it's really called. Still better looking that heatmap.

Neat...I just wish there was an easy way to give the basemap library a 3-item tuple with (lat, lon, value) and have it automatically do a heat map and not just plot the points. That is an option in some software.

I have something of the sort. Give it later, lon and a value, it performs cubic interpolation and forms a heatmap. Some of the code is specific to my application but I'm sure you can reuse.

https://github.com/rraks/sigcatch

Open to contributions.

Thank you! I'll look.

basemap (like matplotlib) has the contourf method. You can calculate your x, y, and z values using matplotlib.mlab.griddata (altho scipy's are faster), and pass the results to it:

https://stackoverflow.com/a/26885815/416626

Thanks for the link

Related: basemap is deprecated, use cartopy instead! https://scitools.org.uk/cartopy/docs/latest/

cartopy's website assumes that I know what I am doing.

There is no good entrypoint into the documentation, it just starts to tell me about projection details. The documentation outline is a list of the code functions with no clear point to start or indication which places might be good to start at.

I also can't find any examples for quick demonstration and trying. (edit: I found them after clicking all the links. Well... it's in the middle of a list of details I'm not interesting in and says 'Using cartopy with matplotlib'. Oh, at the bottom there is a link with more examples which I could find neither in the index nor in the outline... urgh.)

basemap's website also has a lot of problems, but it is a lot easier to use and we are lazy.

I used to just look for something similar in the gallery: https://scitools.org.uk/cartopy/docs/latest/gallery/ but the docs definitely do assume you know about map projections.

The examples definitely improved a lot in the last year. When I first looked at it there were very few and some of them were logos, not maps.

But again, it is not very obvious that the gallery includes sample code, and that you have to click the pictures to get there.

I know about projections, I just don't care enough for that to be the first thing I see. When I start out I just want to make a basic map, fast. (edit: That's actually typical of many open source projects: They start by telling you about all the fancy tech behind it and let you figure out for yourself how to reach your goal/what goal you can actually reach with it.)

Well, enough of whining about the page to probably unrelated people. I actually found it a sign of trustworthyness if a project page looks chaotic and full of bad copy. It means they're not a startup and/or trying to sell something, usually. (This is just too much though, it is not just not new and fancy with no calls to action and pop-ups for mailing lists, it actively tries to hinder you in doing what you came to do.)

I feel like these days we should be creating more user-friendly graphs - both for the creator of the graph and for the reader. I created this graph of the Atmospheric concentrations of CO2 earlier today (https://kyso.io/KyleOS/atmospheric-co2-concentrations#code=h...). Would love to see the OP's heatmap recreated with plotly.

If you're making heatmaps with GIS data, give some critical thought to the interpretation of the map to make sure you're not just doing this: https://xkcd.com/1138/

This is really cool. Thank you for sharing.

you're welcome

Reply on news.ycombinator.com