Hacker News Reader

Looks awesome, the work on performance especially is much appreciated :D I would love to be able to get flame graphs on my spark cluster per node too - any ideas on how exactly FB does this?

View on HN

Looks they are using the same library from Presto. You would just add the Java agent line to your Spark config i.e. spark.executor.extraJavaOptions.

https://github.com/prestodb/presto/issues/4004

Although you have to distribute the library across to all of the data nodes (if using YARN) or Spark nodes (if in standalone).

Thanks! I was going to ask where I might find it since Google results were a little thin on the ground.

I knocked up a quick Ansible script this morning. It is _very_ rough right now and is more like 'here's what you could do' (and you have to compile perf-map-agent beforehand), but I got it working on a small cluster: https://github.com/falloutdurham/spark-flame

Reply on news.ycombinator.com