mikeshi42

Hey that's a good point on the link! Not something I can change now unfortunately, I was hoping having it near the top of the text post would help too for those that wanted to dig in more :)

That being said - as you've mentioned so many different "store tons of data" apps have been enabled from ClickHouse. Observability is at a point where it's in the same category of: ClickHouse can store a ton of data, OTel can help you collect/process it, and now we just need that analytics user experience layer to present it to the engineers that need an intuitive way to dive in to it all.

imo cursor's small context window on the $20 plan is what kills their agent mode. Try claude code or anything that lets you use larger context windows (I think cursor has a tier you can pay for now?). claude code regularly deletes and corrects code and LLMs are very much capable of that today.

Can confirm the workbench does with `claude-sonnet-4-20250514` returns Biden (with a claimed April 2024 cutoff date) while Chat returns Trump (as encoded in the system prompt, with no cutoff date mention). Interesting

I don't think there's a right answer here, but there's definitely a point where your code contributions are much lower leverage than for example trying to recruit the next set of critical engineers, working on the technical roadmap to keep ahead of the competition, or making sure the engineering org is aligned with the rest of the company.

Any lines of code the VP/CTO could write, could likely be written by someone else on their team (and their team's quality could be even better) - but all the other items I listed is likely only something the VP/CTO could do the best at in the company. It's quite a rational decision to largely give up hands-on technical work for what's more important for your team and company.

agreed! or these days just aggregate them directly as if they were metrics

Honestly sounds like good advice for almost anything new/creative :)

(not op) curious what you find frustrating about it?

author here, just wanted to chime in that I really appreciate the kind words about what we've been building :)

This looks pretty cool, I love seeing more clickhouse-native logging platforms springing up! It's a surprisingly underrated platform to build on when I talk to other engineers.

I'm one of those authors of an existing log viewer (hyperdx) and was curious if we were one of those platforms that didn't fit your needs? Always love learning what use cases inspire different approaches.

I do recall doing the same - partially for myself to remember all the meal hacks coworkers gave me! Putting those packaged hardboiled eggs and seaweed snacks into shin ramen is still memorable to this day.

Doesn't the last layer output a variable-size vector based on seq length? It'd take a bit of hacking to get it to be a semantic vector.

Additionally, that vector is trained to predict next token as opposed to semantic similarity. I'd assume models trained specifically towards semantic similarity would outperform (I have not bothered comparing both in the past - MMTEB seems to imply so)

At that point - it seems quite reasonable to just pass the sentence into an embedding model.

Something needs to generate the document embeddings since the LLM itself won't

Caddy isn't just for local docker-based mac development - I use it for any small project hosted on a linux VPS for example.

You want to be careful of async inserts as it's in-memory buffered as well and is not replicated afaik. So you can lose a bit of durability when using it (or you make sure your upstream pipeline can retry).

It is really nice in a pinch though.

(shameless plug) maybe it was us?

https://github.com/hyperdxio/hyperdx

Surprisingly I did as well, at least for me it looks like I just misread it (some sort of human bias?) Interesting to be n=3.

I didn’t downvote but s3 does have low latency offerings (express). Which has reasonable latency compared to EFS iirc. I’d be shocked if it was as popular as the other higher latency s3 tiers though.

Matt Levine recently covered how it's much more lucrative/easy to sue Chipotle for skimping out on their portions from the investor/security fraud angle than it is the customer angle [1]

Unfortunate how that's just how the system works.

[1] https://archive.ph/mhEQA

wow gmail drive is a walk down memory lane :) really is amazing how far we've come since then!

Yup! Since we're built on Otel, we suggest you can use the HTTP OTLP endpoint, which is yourdomain:4318/v1/logs, we have a quick blog post on that on how to curl a payload over: https://www.hyperdx.io/blog/testing-sending-opentelemetry-ev...

But you can also look at example payloads in the otel repo: https://github.com/open-telemetry/opentelemetry-proto/blob/m...

Great call out on CF workers, will make sure we get that updated :)

Yup it'd work if the Clickhouse HTTP interface is preserved, along with the CH-specific functions/settings we leverage. (I'm not sure how deviated your fork is from CH and which version it's based on)

Proton looks like a neat queue/db combo - I'm going to have to dive in deeper some time

Yup! We're agnostic to the storage medium underneath as Clickhouse's interface doesn't change based on the disk layout backing the table. In our hosted version we use S3 as well.

They have their docs on that setup here: https://clickhouse.com/docs/en/guides/separation-storage-com...

Thank you, really appreciate feedback like that! :D

yes! the full stack includes our recommended schema which has the indexes set up - it's a drop in replacement for anything that would ingest Otel-based telemetry! If you already have Promtail setup - you might want to set up a collector or tweak the existing collector to take in Promtail via the Otel Loki Receiver: https://github.com/open-telemetry/opentelemetry-collector-co...

Overall it doesn't sound very hard to me!

thank you! means a lot coming from you ;)

Speaking of the system tables - it's awesome how much telemetry is saved in there that helps us build a really powerful preset clickhouse monitoring dashboard (heavily inspired from the built-in clickhouse one of course). We figured that alone is quite useful for teams running any Clickhouse instance and want better insights into what's going on.

You'd generally add an index to your logs in Clickhouse to do searching (via ngram or token bloom filters typically: https://clickhouse.com/docs/en/optimize/skipping-indexes#blo...). There's other ways of indexing as well but that's generally the best for full text search. We use token bloom filter indexes today and find them quite effective (it can skip whole chunks of logs due to the bloom filter being able to say that a word did not appear in the chunk of logs).

Indeed Loki is incredibly slow - Clickhouse is deployed for logging at scale (ex. trip.com is running a 50pb logging solution that allowed them to 4x their old ES cluster volume while also running queries 4-30x faster)

Totally fair! Here's a few on the top of my head, they're mostly about Elastic really but of course Kibana is only really useful on ES:

1. At my last job we were running some of the largest elastic clusters of our hyperscaler's cloud - elastic is slow, expensive and finicky to operate at decent scale. We've found the exact opposite with Clickhouse, it's fast, easy to operate, and supports things like S3-backed storage directly in their open source product. As an example, Uber switched from Elastic -> Clickhouse and halved their infra footprint while increasing volume.

2. Elastic is tricky to manage, field type conflicts come up super common at scale and are annoying to deconflict. Clickhouse is a lot more flexible in its schema to avoid those problems (and give you knobs to fine tune performance at a more granular level with their indexes/schemas)

3. We allow for both SQL and Lucene, both are relatively "standard" languages that engineers are likely already familiar in one way or another. Compared to elastic moving to ES|QL, another vendor-specific language that will be difficult to onboard to. The last thing you want during an incident is trying to recall vendor-specific languages for querying that critical data!

tl;dr - we try to make it easy to "do observability" on what we think is the best DB for observability today (Clickhouse), analogous to what Kibana did for ES.

If you're using the Grafana Clickhouse plugin - a number of things we do differently to them today:

- We support Lucene-based search, which means it's a lot easier to find the events you're looking for without needing to break into verbose SQL search. (Column/property/full text search are all super easy).

- We're optimized exclusively for Clickhouse, which means we do a number of things to optimize the queries we run and that returns you a nice performance boost (we see a 2x perf boost, but this will vary a ton of data and queries). For example we allow you to do a search on a subset of columns (so the search is performant), and then click in and expand an entire row of interest on-demand (so we only do a SELECT * for a single row). This is also a much nicer DX than needing to specify every column you might need. We have a few other optimizations as well.

- We have (what we think is) a nice chart builder - so you don't need to mess with template variables and macros to build a chart, but still lets you escape hatch out into SQL for the important bits.

- We think our event deltas feature for analyzing traces is pretty neat - afaik this isn't something you get in Grafana. - You can live tail events, I don't think Grafana for Clickhouse has that (I'm a big fan of tail -f)

Overall we focus on trying to bring an easy-to-use high-cardinality observability experience to Clickhouse, whereas Grafana seems to focus more on a highly SQL-dependent dashboard building experience (which has its own advantages of course).

edit: fixed line breaks!

Thank you Max! It's awesome to hear that :)

Our v1 is completely built on Clickhouse! So v2 is making it more widely compatible to Clickhouse installations that aren't tied to our ingestion pipeline and schema. So if you're already on Clickhouse for obseravbility today, or have a preferred way of shipping data in, you can use us on top of Clickhouse now without throwing away your existing work.

We're essentially making our existing product a lot easier to deploy into more complex observability setups based on Clickhouse - while also shipping a few new capabilities like SQL and event deltas while we're at it!