Show HN: UI for Apache Kafka | Hacker News Reader

Show HN: UI for Apache Kafka

github.com

158 pointsgermanosin53 comments

Looks pretty nice. The codebase is half typescript and half Java? Guessing that is because Kafka is written in Java.

Another question: how might this compare to the Confluent web GUI? Obviously this is open source which is great.

At a first glance this seems to be an api written in Java using Spring Boot in a reactive style with Webflux. The api wraps the official Java Kafka client and some other libraries.

The frontend is a React app in Typescript.

Hi, yes exactly. Backend is on Java + spring boot webflux, because of Kafka most mature client and frontend react + typescript.

Btw I have an article comparing it with confluent web GUI https://towardsdatascience.com/overview-of-ui-tools-for-moni...

You should have mentioned that you work at Provectus in this article. Feels pretty sneaky to claim to do an impartial review, but provide no negative critique of the one made by your company…

This is indeed a very biased comparision where some features in the comparision table are just claimed to be not supported by other (open source) solutions. Even after reaching out multiple times the article is not updated accordingly. So this has to be considered a sneaky marketing page to be honest.

I might be biased as I'm one of the Kowl authors but I can definetely tell that Lenses is affected negatively the most. Imo it is one of the most complete paid solutions and got the worst marks basically.

This is mentioned in my medium profile. If you think that my comparison is not objective feel free to mention it on comment there)

For example, it seems a bit odd that all of your "no" answers in a feature matrix are labelled "roadmap", but every other tool just gets "no".

> feel free to drive engagement on my post

https://lenses.io/ Kafka UI is pretty solid. Nice to see an open source alternative here.

Can't get over the amount of corporate memphis on that page.

> corporate memphis

Links for the lazy. The GP page (https://lenses.io) certainly has a lot of good examples.

https://en.wikipedia.org/wiki/Corporate_Memphis

https://t-artmagazine.com/what-is-corporate-memphis-and-why-...

Haha thank you. Had no idea.

Me too hahaha

There are several alternatives posted here already, but after comparing several ones maybe a year ago I found Kowl very nice: Nice UI, backend written in Go (quick startup, low memory usage), viewing and filtering of messages, support for schema registry, comes with a Docker image, usable locally and Helm chart for Kubernetes.

https://github.com/cloudhut/kowl

kowl is awesome. The only missing feature I've found is that when looking at groups, only lag is shown but not the actual position.

In the newer versions you can see the actual offset in the group details page :-)

Plus one for kowl. Their support is also amazing.

Neat! Just be wary around licensing, we’ve been using a similar management tool for a couple years https://github.com/yahoo/CMAK which was originally Kafka manager but they were forced to rename it.

There was a whole wave of renamings in the past two years, it looked like Apache Foundation just carpet bombed everything with "Kafka" in the name.

We had a discussion with Apache team member according the name. Their main concern was that name should contain not Kafka but Apache Kafka. So they are ok with our current product name "UI for Apache Kafka"

Where I work we use Confluent Cloud who has their own proprietary UI. I've always found it to be lacking, hard to use and not very good. We substituted it with AKHQ https://akhq.io/ , which is miles ahead of anything I've seen. The main issue with it are the interesting UX decisions that requires learning. For example, a lot of links require a double click, which isn't a common behavior in Web Apps. Besides that, it's absolutely wonderful and goes beyond just Kafka. We use Kafka Connect very heavily and AKHQ seems to implement the full CRUD for Connect.

UI for Apache also provides full support for Kafka Connect

Thanks for posting. Does it support AWS MSK?

Sure, This fully support AWS MSK including AWS IAM authentication, Glue schema registry and AWS Kafka Connect

Nice. Is there are documentation on how to set it up for MSK?

You could start with documentation here https://github.com/provectus/kafka-ui/blob/master/guides/AWS.... Feel free to create an issue if you have any questions.

https://lenses.io/cloud/aws-msk/

I rolled out kPow initially for my team of 10 to help us manage and monitor our clusters.

We’ve now rolled it out to our 400+ engineers who use it every day to help them gain visibility and insight to their data and topics, manage offsets, amount other things. It handles HUGE clusters with no issue, and the RBAC features are great.

I’ve tried them all. While there are some ok alternatives, I honestly find them toys compared to kPow. Do yourself a favor and check it out!

We use Kafdrop for viewing topics/messages/info and it's really nice. It's read-only though, so if you need to configure a topic or push a message you have to use something else.

https://github.com/HomeAdvisor/Kafdrop

This is the more up-to-date repo: https://github.com/obsidiandynamics/kafdrop

Kafdrop is great, especially if you're already running everything else in Docker, just add a few lines to your docker-compose and get a free web UI for troubleshooting/introspecting.

What is the difference between this and https://github.com/yahoo/CMAK

There's a lot great and developing tools for Kafka right now. CMAK doesn't seem that active, and I believe it was one of the tools we quickly gave up on, because it was a little difficult to get going.

Currently we're using Kowl, and we're looking into Cruise Control.

One limiting factor is that many of these tools doesn't support authentication and RBAC, Kowl does, and that has been a requirement for at least some of our projects, where not everyone is allowed to view certain topics.

I'm curious what makes you look into other solutions than Kowl then? :). We are happy to add anything that is desired in the community. Now that Kafka connect support is also there we will most likely just expand the existing features (e.g. adding, editing schemas etc.).

Disclaimer: I'm one of the Kowl authors. I do not intend to hijack that post for Kowl content though!

For ourselves, nothing really. For a customer: They really don't want to pay the licens once it is required, and they need ActiveDirectory authentication or KeyCloak. I've already been in contact with someone working on Kowl, and we're just waiting on OIDC support.

Really we just want an alternative to Kowl, if a customer asks for suggestions. For our own use, we plan on keeping Kowl.

UI for Apache Kafka supports OIDC (https://github.com/provectus/kafka-ui/blob/master/guides/SSO...) and basic authentication (https://github.com/provectus/kafka-ui/blob/683ab231f436893b3...)

Yahoo CMAK is mostly focused on ops, UI for Apache Kafka is mostly for devs, providing Kafka ecosystem integration and observability.

The UI looks great. I wanted to try it but am unable to run it in mac with m1 chip. Is there a docker image for arm based processors as well?

Could you please create an issue in the repo with detailed description / logs. By design it should work.

I'm using kafdrop but modify it a bit. Reasons: - the front page is very slow if there are a lot of topics (the kafka cluster I'm using has 6k topics). It's because kafdrop tries to get a lot of information of every topic even though it doesn't display them on the screen. I modified it like this: after display 6k topic names, I send ajax requests to get info (# of partitions, start & end offsets of each partition, number of messages in each partition) of 20 topics each. - kafdrop doesn't have the feature to send a message to a topic. I added a text area below each topic name so that I can type the mesaage there and send. - If there's a new message, kafdrop cannot tell me which partition that new message goes to. So I added this feature: suppose that a topic has 3 partitions with 3, 4 and 5 messages. If there are some new messages coming to that topic, I can refresh that topic in the UI and the UI will show something like partition 2: 3 -> 4 messages. So I know that 1 new message goes to partition 2.

I like the multi-cluster management feature of this project. Unfortunately I don't know react (angular + typescript is ok with me), so I'm wondering if I should try this one knowing that I cannot modify it. The reasons I modified kafdrop are also the features I like to see in a new Kafka UI.

See https://kpow.io - if you would like a trial license let me know. 6k topics is fine, might need slight tuning which is easily done (6k groups slightly more complicated).

A few pieces of feedback.

Positive:

  - Multi-cluster support is a great feature! That's the main thing missing from Kowl, in my opinion
  - The UI feels clean
  - Message streaming feels smooth, and the search feature is a great addition. Also love the ability to see a message for a given partition/offset, and that JSON messages are nicely rendered as JSON!

Negative/Feature Requests:

  - The CleanupPolicy logic doesn't handle "delete,compact" (I see bug number 829 addressed a slightly related issue, maybe that logic could be expanded [1])
  - When viewing Cluster -> Topic -> Consumers, it seems like far more is being loaded than just consumer groups for that particular topic. It takes much longer to load than, e.g., Kowl does for pulling the same information
  - Similar feedback for when clicking on an individual consumer group (example URL path: "/ui/clusters/dev/consumer-groups/my-group"). It takes a very long time to load
  - If any permissions issues are encountered while loading the individual consumer group, the entire request fails. That wouldn't necessarily be an issue, but it is when you're loading all consumer groups, and not just the one requested
  - I don't see any docs on how to access a topic that is secured with certificates (Kowl's relevant config [2])
  - If a schema registry uses a self-signed certificate (or one signed by a corporation's CA), there's no way to pass the certificate through a config or skip validation (Kowl's relevant config [3]). As it is, you get a 500 error in the API call, but the UI doesn't show an error

Overall, it seems like a fair bit of thought went into this! But there are a few quirks that make it very tough to use in a corporate environment with (perhaps excessively) large Kafka clusters with hundreds/thousands of topics, or secured with internal certificates

1. https://github.com/provectus/kafka-ui/blob/master/kafka-ui-a...

2. https://github.com/cloudhut/kowl/blob/10776b81e45abfa716548d...

3. https://github.com/cloudhut/kowl/blob/10776b81e45abfa716548d...

  Hi, db3pt0
  Thanks for detailed feedback!

  - The CleanupPolicy
  fixed issue 925 (https://github.com/provectus/kafka-ui/issues/925)

  - When viewing Cluster -> Topic -> Consumers, it seems like far more is being loaded than just consumer groups for that particular topic. 
  It takes much longer to load than, e.g., Kowl does for pulling the same information
  There is only one way to filter consumer groups by topic, get all consumer groups, enrich it with current members, committed offsets, and then filter them by topic.
  If you have a lot of consumer groups, this might take time to do it. We are thinking on this issue, and will try to improve it in the next versions.
  (https://github.com/provectus/kafka-ui/issues/927)

  - Similar feedback for when clicking on an individual consumer group (example URL path: "/ui/clusters/dev/consumer-groups/my-group"). It takes a very long time to load
 This looks rather strange. For single consumer we are getting consumer group description and then enrich it with topic info (end offsets). This works fast enough.

  - If any permissions issues are encountered while loading the individual consumer group, the entire request fails. That wouldn't necessarily be an issue, but it is when you're loading all consumer groups, and not just the one requested
  Thx for describing this. We created issue for this. (https://github.com/provectus/kafka-ui/issues/928)


  - I don't see any docs on how to access a topic that is secured with certificates (Kowl's relevant config [2])
  You have to pass ssl config stores to kafka client (we'll add docs. in the next versions https://github.com/provectus/kafka-ui/issues/929)

  KAFKA_CLUSTERS_0_PROPERTIES_SSL_KEYSTORE_LOCATION = /var/private/ssl/kafka.server.keystore.jks
  KAFKA_CLUSTERS_0_PROPERTIES_SSL_KEYSTORE_PASSWORD = test1234
  KAFKA_CLUSTERS_0_PROPERTIES_SSL_KEY_PASSWORD = test1234
  KAFKA_CLUSTERS_0_PROPERTIES_SSL_TRUSTSTORE_LOCATION = /var/private/ssl/kafka.server.truststore.jks
  KAFKA_CLUSTERS_0_PROPERTIES_SSL_TRUSTSTORE_PASSWORD = test1234

  - If a schema registry uses a self-signed certificate (or one signed by a corporation's CA), there's no way to pass the certificate through a config or skip validation (Kowl's relevant config [3]). As it is, you get a 500 error in the API call, but the UI doesn't show an error

  Same as above (https://github.com/provectus/kafka-ui/issues/930)
  KAFKA_CLUSTERS_0_PROPERTIES_SSL_TRUSTSTORE_LOCATION = /var/private/ssl/kafka.server.truststore.jks
  KAFKA_CLUSTERS_0_PROPERTIES_SSL_TRUSTSTORE_PASSWORD = test1234

Oh, cool! We are already using this tool and happy to it here

https://kpow.io - we build kPow for Apache Kafka, a drop-in replacement for Confluent Control Center + much more.

https://kpow.io/features

- Multi-Cluster, Multi-Schema, Multi-Connect.

- Multi-Tenancy, RBAC, all the Authz - LDAP, SAML, etc.

- Very fast streaming multi-topic search (~1M messages / minute)

- Built in JQ search filters (we implemented a subset of JQ in the browser and JVM)

- Live Kafka Streams / Consumer Group topology visualisation (including RocksDB metrics for state stores, etc)

- Offset management, resetting, scheduled mutations.

- Consumer Group read and Topic write freshness / n-minutes inactivity metrics.

- Heaps of admin functions like Staged Mutations, Temporary RBAC policies, Data Masking for PII, etc.

- Prometheus / OpenMetrics endpoints.

- kREPL (slightly mad programmatic interface to Kafka)

- 7k commits, 100k docker pulls, 97.4% Clojure.

- Unlimited users, unlimited features.

- Commercially supported, annual licenses or available as hourly-metered (0.16c/hr) on the AWS Marketplace or Red Hat Marketplace.

Multi-Tenancy is probably worth a link:

https://kpow.io/how-to/manage-kafka-visibility-with-multi-te...

kPow gives you the ability to restrict visibility of Kafka resources down to a group or topic level by user-role, then builds completely internally consistent synthetic cluster views for each tenant.

We have a number of large enterprise customers who requested multi-tenancy support, it sounded tough at first but due to how kPow is built (we compute our own metrics / no JMX) and the fact that we use Clojure in both the front/back end it turned out to be fairly straight forward to implement.

Is the monitoring functionality independent off the cluster it is monitoring being up to see monitoring?

Control centre eats its own dog food. All its data is on topics. That means if your cluster is unhealthy you can see healthy stats until the control centre consumers have caught up. Which means if you have a lot of data and load, control centre can start lagging and in one case I had some very hard debugging wondering why control centre saying the cluster was unhealthy but the cluster was up. It turned out control centre was lagging by 24 hrs so I was seeing the previous days outage.

Confluent recommended putting control centre on its own cluster for this very reason. One that isn’t cheap, a minimal prod cluster is 3 brokers+zookeepers and secondly what then monitors your monitoring cluster.

Yes, in two ways.

First - kPow also stores all telemetry in internal topics (and also keeps an audit-log in another topic for data governance). Setting up multi-cluster is fairly easy and kPow stores data in the first configured cluster - so like Confluent's suggestion you could have a dedicated kPow cluster. We really only recommend this if/when you get up to a large number of connected Kafka resources (Cluster, Schema, or Connect) and/or you have cluster/s that are very busy with other work.

Second - from v85 (next week) our v3 compute engine make kPow monitoring functionally independent of topic load / performance of the underlying cluster. We do still persist to internal topics for recovery, etc, but live stats and control are only dependent on your Kafka cluster responding to AdminClient requests (a handful every minute).

If you would like a trial license or guided tour, just say.

spoken to a few companies that have started using this internally - bit of a game changer apparently, kudos

kRepl is mind blowing. Our engineers have used it very effectively to help diagnose issues in production.

Cool, we're using this in a couple projects, awesome tool!

Love it, just got on Kafka a few days ago

I use the vscode plugin. As a matter of fact I seem to be using vscode for more and more admin tasks including kube management.

Reply on news.ycombinator.com