Show HN: Delight, a free hosted cross-platform Spark UI and Spark History Server

Hello HN community,

I’m JY, co-founder of Data Mechanics (YC S19, https://www.datamechanics.co). But this post is NOT about startup core product (a managed Spark platform, deployed on a k8s cluster in our customers cloud account, see our HN Launch https://news.ycombinator.com/item?id=23142831).

This post is about a free and partly open-source monitoring tool for Apache Spark that we just released.

It works by installing an open-source Spark agent (https://github.com/datamechanics/delight) on your Spark infrastructure — whatever it is: commercial or open-source, on Kubernetes or on YARN, in the cloud or on-premise.

This agent streams event metrics from Spark (metadata about your Spark applications) to our backend, which then serves a dashboard listing the Spark applications, and giving you access to the Spark UI (Spark History Server) for each of them. The blog post has a lot more details about the architecture and security of it.

This release is just a first milestone for us, in our next release in January we will add new screens to gradually replace the Spark UI with a new monitoring view, you can see a glimpse of it at the GIF at the bottom of this page (https://www.datamechanics.co/delight)

We’d love your feedback about it — is it easy to install and use? What would you like to see in following releases? Thanks so much! JY