mikeshi42

Hrm while we aren't a 1:1 Kibana replacement today (we're not apples to apples since Kibana is locked into Elastic, whereas we're on Clickhouse) - I don't think we're too far off with our UI-based filters, Lucene filter language, and timestamp filtering/sorting/live tail.

There is a setup modal (which is intended to be set up once by the data owner, similar to maybe how you'd set up some indexes in Elastic) - and afterwards the experience is similar IMO. If you're open to sharing more I'd love to learn more mike@hyperdx.io or if you want to open an issue/join our discord.

Ahh yeah fwiw it wasn't _intended_ to be a dig at the open source status of Kibana - but rather we're open source and building on top of Clickhouse.

On the commercial OSS side of things - I suspect the trend there is more nuanced than all OSS companies being suspect to the same problem, but rather companies that generally solve a "behind the API" problem are more susceptible to problems of cloud vendors taking their code and competing with them commercially (ex. if you're a DB like Redis, Mongo, Elastic - or a CLI like Terraform). We're building an end-user experience (more like Gitlab) - where experience differentiation matters a lot more than simple infrastructure hosting, something AWS is not particularly well suited at competing on!

It's been 3 years of Gitlab post-IPO and they're still MIT, and that's the boat we're on as well :)

Do you have any plans to be able to take TypeScript types to do type optimizations? That sounds like a bonus to me to be able to have easier implementation and faster run times.

In AWS iirc NVMe max out at 2GB/s - I'm not sure why that's the case. I know there were issues with the PCIe controller in the past being the bottleneck, but I suspect there's something more to it than that.

Agreed. At the end of the day - the end user of the software probably wants something other than technical debt reduction, so it's not surprising impact and LOC can roughly be correlated.

Taking the LOC metric too far, in either direction, is trying to read too much into a single metric.

> Somewhat hilariously I was a manager during Covid, and as such I saw the metrics which showed that every worker group in the city I was working for was happier and more productive.

I'm curious how productivity was measured for that. I assume happiness was self-assessed, but I'm also surprised that'd be up given how much of a rollercoaster a time that was just in general.

> I can't evaluate the correctness of a choice without either knowing the motivation for it, or by learning the problem domain well enough to identify and make the choice myself - at which point the convenience of the AI solution is abnegated because I may as well have written it myself.

I think I usually accept code that is in the latter - the convenience is I did not need to spend any real energy implementing the solution or thinking too deeply about it. Sometimes the LLM will produce a more interesting approach that I did not consider initially but is actually nicer than what I wanted to do (afaik). Often it does what I want or something similar enough to what I would've written - just that it can do it instantly instead of me manually typing, doc searching, adding types, and correcting the code. If it does something weird that I don't agree with, I instead modify the prompt to align closer to the solution I had in mind. Much like Google, sometimes the first query does not do the trick and a query reformulation is required.

I wouldn't trust an LLM to write large chunks of code that I wouldn't have been able to write/figure out myself - it's more of a coding accelerant than an autonomous engineer for me (maybe that's where our PoVs diverged initially).

I suspect the similarity with PRs is that when I'm assigned a PR, I generally have enough knowledge about the proposed modification to have an opinion on how it should be done and the benefits/drawbacks of each implementation. The divergence from a PR is that I can ask the LLM for a modification of approach with just a few seconds and continue to ask for changes until I'm satisfied (so it doesn't matter if the LLM chose an approach I don't understand - I can just ask it to align with the approach I believe is optimal).

I'm assuming by bullshitting you mean differentiating between LLM hallucinations and a human with low confidence in their code.

I've found that LLMs do sometimes acknowledge hallucinations. But really the check is much easier than a PR/questioning an author - just run the code given by the copilot and check that it works, just as if you typed it yourself.

I've done some porting between Python and JS based on Tensorflow in the past - and I suspect the poor ergonomics in JS for math/lists would probably ruin the experience a good amount.

Perhaps something like Pyodide can bridge the gap and make it easier to bring into the browser as well.

I read this exact reply N days ago... okay exact is stretching it but you get the point ;)

https://news.ycombinator.com/item?id=41683174

I agree the cat is out of the bag, but GANs do not work like that. One of the common failure modes in training a GAN is that the discriminator gets too powerful too quickly and the generator then can no longer learn.

Hard to say anything is impossible off of one point - but discrimination afaik is generally seen as the easier problem of the two, given you only need to give a binary output as opposed to a continuous one.

Curious if https://github.com/hyperdxio/hyperdx passes the bar for looking good!

(disclaimer: I'm a maintainer that's done a good chunk of the UI - but I also love candid feedback)

How are you guys using sqlite in dev instead of clickhouse? (Afaik there's a good bit of difference between the two dialects so I'm surprised it's possible without hurting dx through one compromise or another)

we've leveraged Clickhouse/S3 to build a cost effective alternative to Datadog at https://hyperdx.io (OSS, so you can self-host as well if you'd like)

We pull in everything across our AWS infra via otel collectors (metrics/logs/traces) and forward to HyperDX (ourselves, w/ storage backed on Clickhouse). You'll find that Clickhouse is a ton more efficient than Elastic when it comes to observability use cases, which helps a lot with keeping costs under control. There's less schema management typically needed on Clickhouse as they have more flexible map types for chaotic structured logs. The Otel collector is also very flexible in adding filtering rules to throw out noisy messages.

would love to hear more on what you mean

This is really exciting since the Otel collector already supports Datadog tracing, adding metrics will cover the big two signals that make it hard for most teams to shift from DD's walled garden. I suspect this will really ease the migration path from closed standards to open for a whole swath of companies now.

Hopefully the PRs get upstreamed quickly without any opposition from Datadog unlike last time.

std disclaimer: I'm building oss obs for hyperdx, so I have a clear horse in the race for companies adopting open standards, just like grafana.

No you're not missing anything, it is that simple. With @opentelemetry/auto-instrumentations-node you can actually get the OTLP exporters "for free" as well if you require the register script.

Logs in JS is really only significant as a wire format + some transport glue code for popular loggers so it can be more or less completely ignored. I do find the Otel metrics model overcomplicated compared to OpenMetrics, though not to say OpenMetrics doesn't have its own set of tradeoffs.

With metrics and logs, I do see the advantage of a single project driving unification of signals as opposed to treating them as silos. With one set of semantic conventions and SDK available, it'll be a lot easier to tie everything together, rather than relying on engineers to do that themselves via timestamps and glancing across 3 different tabs.

I think the OP author bias here is that Sentry only really cares about the Otel instrumentation libs for tracing to power their APM product, and everything else is superfluous (hence the blog). I help build HyperDX, where we care about unifying metrics/logs along with traces, and having it all in one ecosystem is quite important for making it work well - and we do not find the rest of it superfluous! (disclaimer: obviously biased in my own ways)

The Otel spec does give leeway for language-specific details, and the SDKs are not as uniform as you'd expect (ex. Java's agent configuration is very different from Node's auto instrumentation). I'm not denying that there's SDK specs to adhere to, but the abstraction complexity in Otel is really from the amount of flexibility they've tried to build into the SDK for better or for worse.

The flexibility benefits vendors (I work for HyperDX, based on otel) - as it allows for a lot of points of extensibility to build a better experience for end users by extending the vanilla SDK functionality. However, it creates a lot of overhead for end-users trying to adopt the "vanilla" SDKs out of the box as there's 5 layers of abstractions that need to be understood before getting things started (which is bad!)

I've only seen the DX of Otel improve over time across the ecosystems they support - so I suspect we'll get there soon enough.

It's a bit confusing but here's my best attempt to explain it:

- Trace events (span events) are intended to be structured events and possibly can have semantic attributes behind them - similar to how spans have semantic attributes. They're great if your team is all bought in on tracing as an organization. They will colocate your span events with your parent span. In practice they have poor searchability/indexing in many tools, so they should only be used if you only intend to use them when you will discover the span first. (Ex. debug info that is only useful to figure out why a span was very slow and you're okay not being easily searchable)

- Log records are plain old logs, they should be structured, but don't have to be, and there isn't a high expectation of structured data, much less semantic attributes. Logs can be easily adopted without buying into tracing.

- Events API, this is an experimental part of Otel, but is intended to be an API that emits logs with the expectation of semantic conventions (and therefore is also structured). Afaik end users are not the intended audience of this API.

Many teams fall along the spectrum of logs vs tracing which is why there's options to do things multiple ways. My personal take is that log records are going to continue to be more flexible than span events as an end-user given the state of current tools.

Disclaimer: I help build hyperdx, we're oss, otel-based observability and we've made product decisions based on the above opinions.

> What I remember hearing from YC is of cases where a partner tells you to pivot, not where they let the founders decide to pivot themselves.

Post-batch you can talk to your partners as often as you want or never again - they stay out of your way unless you explicitly reach out. They make this quite clear during the batch as well.

During the batch you do talk to your partners as part of the structure. They do give (really good!) advice, but you're always told it's your company. I can see them giving advice based on their experience on where you might want to dig instead, but I've never heard a founder being forced down a direction they didn't want to go. After all, why would that be the optimal path?

This is incredibly common - I've heard a company end up rearchitecting their instance type choices due to Datadog billing on a per-node basis (with some peak usage billing shenanigans). Their business model unfortunately encourages some very specific architectures which doesn't work for everyone.

We're one of those OSS alternatives (HyperDX) - built on Clickhouse. While I can't say it's stupid simple to scale Clickhouse (because anything stateful is inherently hard to scale), it's orders of magnitude easier than other platforms like Elastic and gives you a lot more flexible tuning options (last co I was at, we ran that at massive scale, it was an absolute handful).

In theory, you can get away with just running a Clickhouse instance purely backed by S3 to get durability + scalability (at the cost of performance of course). It all depends on what scale you're running at and the HA/performance requirements you need.

I'm wondering when's the last time you tried OpenTelemetry - and which language it was in? I'm not going to say it's super mature (it's not) - but I think it's come a long way from being a ton of manual setup and it's more akin to SDKs available commercially. Admittedly we (HyperDX) do offer some wrapped OpenTelemetry SDKs ourselves to users to make it even easier - but I think the base Otel instrumentation is easy enough as it is.

Oh yeah that's a great idea - just set it up once in minikube and use it just like you were debugging something in your production k8s stack. As much as I love k9s, sometimes I just need a trace to debug something between services

HyperDX (YC S22) | Founding Backend Engineer | Hybrid SF/San Mateo | $115k - 180k

We're building an open source, dev friendly observability tool (think Datadog, but something developers actually love to use and companies can actually afford).

We're in the intersection of needing to build rock solid infrastructure ingesting TBs of data, searching it incredibly quickly and scalably, and layering on top a buttery smooth DX from our language-specific SDKs, APIs and web app.

We're super early and hiring our first founding engineer. We already have a cloud product customers pay for and love, loads of runway regardless of the wider economy, 5k+ Github stars, and tons of hard technical problems.

The vast majority of our work is open source, so you can get a sense of what you'd be working with here: https://github.com/hyperdxio/hyperdx

Our job listing is here as well: https://www.ycombinator.com/companies/hyperdx/jobs/1Ja2psR-f...

Come by our discord as well if you just want to talk shop: https://hyperdx.io/discord

I'm Mike, one of the cofounders. If you love shipping quickly and want to help us build from the ground up an open source developer tool (that devs won't hate when they're on-call), let me know! mike [at] hyperdx.io

Biased as a founder in the space [1] but I think with OpenTelemetry + OSS extensible observability tooling, the holy grail of one tool is more realizable than ever.

Vendor lock in with Otel now is hopefully a thing of the past - but now that more obs solutions are going open source, hopefully it's not necessarily true that one tool would be mediocre over all use cases (since DD and the likes are inherently limited by their own engineering teams, vs OSS products can have community/customer contributions to improve the surface area over time on top of the core maintainer's work).

[1] https://github.com/hyperdxio/hyperdx

HyperDX (YC S22) | Founding Engineer | SF Bay Area HQ | REMOTE

We're building an open source, dev friendly observability tool (think Datadog, but something developers actually love to use and companies can actually afford).

We're in the intersection of needing to build rock solid infrastructure ingesting TBs of data, searching it incredibly quickly and scalably, and layering on top a buttery smooth DX from our language-specific SDKs, APIs and web app.

We're super early and hiring our first founding engineer. We already have a cloud product customers pay for and love, loads of runway regardless of the wider economy, 5k+ Github stars weeks after our OSS launch, and tons of hard technical problems.

The vast majority of our work is open source, so you can get a sense of what you'd be working with here: https://github.com/hyperdxio/hyperdx

Our job listing is here as well: https://www.ycombinator.com/companies/hyperdx/jobs/zFXTbzl-f...

Come by our discord as well if you just want to talk shop: https://discord.gg/FErRRKU78j

I'm Mike, one of the cofounders. If you love shipping quickly and want to help us build from the ground up an open source developer tool (that devs won't hate when they're on-call), let me know! mike [at] hyperdx.io

Oh huh! What would you do with span start independent of stops?

A tangent on logcat - local observability to me is a really intriguing area, I think there's a story of Otel for local as well if someone can build a good enough local DX for consuming them (we've been told a number of times about this https://github.com/hyperdxio/hyperdx/issues/7 as an example)

I think it's a huge shame, as Rust is one of the few languages that tracing is effectively native in, and it's shocking that the defacto standard of tracing transport doesn't interop well.

Admittedly, I think part of it is just where Rust is on the language adoption curve and where Otel is in its project lifecycle as well. Writing your own subscriber might not be a terrible idea, we internally end up needing to do similar things as we find limitations/bugs that we can't wait for upstream to fix (but we're a vendor and that makes more sense for us than end users!)