mikeshi42

No worries - excited to hear what you think! Feel free to drop by our discord if you run into any issues or have any other feedback as well :)

Agreed! It's per-host pricing can obliterate budgets if you use a fleet of small instances (which is crazy to me their pricing dictates your infra...)

Would love to have you check us out! Let me know if you run into any issues - feel free to hop on our discord as well :)

Awesome, let me know what you think when you get a chance to take a look!

Yes! It should definitely be thoughtful about what you log and how you expect to use it. My biggest gripe with logs is often people writing them never think about "how would I use this when things are on fire?" and tend to log useless information or fail to tag them in ways that are actually searchable.

Tagging the right IDs are a huge thing - customer X is saying their instance is really slow, but if none of your logs let you link service performance to customer X, your telemetry you're paying for is absolutely useless!

You have an ally in me on this one :) I'm hoping given a bit more time we get to write things like this - practical observability from the perspective of a dev, as opposed to the SRE angle that I think is well covered. Feel free to join us on discord btw if you want to chat more - I (for better/worse) love musing about these things :)

Oh yeah browsers are really just another service (and that's what we try to treat it as, as well!) and it's really the same set of questions you'd ask of any service, but for some reason the tooling completely stops either at the frontend or at the backend.

As for monospace font - feedback received! Is there a particular section you think is too overwhelming? (search page, nav bar, etc.) We've been thinking of how can we balance between the ease of monospace for reading instead of having it literally the default on every UI surface :P

Ah yeah the easiest way is probably using the OpenTelemetry collector to set up a process to pull your logs out of jounrnald and send them via otel logs to HyperDX (or anywhere else that speaks otel) - the docs might be a bit tricky to go around depending on your familiarity with OpenTelemetry but this is what you'd be looking for:

https://github.com/open-telemetry/opentelemetry-collector-co...

Happy to dive more into the discord too if you'd like!

It's a pretty scattered landscape with everyone wanting something slightly different, but everyone has likely heard of Datadog at one point or another (whether they wanted to or not... but that's another story).

It becomes convenient short-hand for what they do (collect logs, metrics, traces, RUM, etc. for engineers to debug).

Though with more characters to write, I'd like to think we have a different take on both how our pricing model works and how easy it should be for an engineer to get started with us :)

Absolutely! You can either self-host the OSS version today, or chat with us (mike@hyperdx.io) directly if you need a managed on-prem solution or any other custom requirements depending on your deployment.

Disclaimer: I'm one of the founders of HyperDX (we're on HN today too!)

I think it depends if you're just looking at price or other factors as well, I'd say the incumbents would be Grafana or Elastic. Though they both have a bit of their pitfalls - Grafana Loki doesn't handle high-cardinality search well (ex. search by ID) versus Elastic is super heavy to have self-hosted and involve some manual schema management. Though they're both very mature offerings and can likely flex to arbitrary workloads.

On the other side of things I'd say Signoz or HyperDX (what we're building) can work as well, though again each tool has its set of UX tradeoffs it's made so far and it really depends on what workflows you care for the most.

Overall we're highly focused on providing solid developer workflows, ex. with HyperDX users can correlate a log to a trace (and vice-versa) really easily in the same UI, we don't silo out features that are commonly needed in a single workflow. You can also search everything from a single panel, whether it's a log, trace, or client-side event, using the same syntax which means there's less to learn.

Feature-to-feature, I'd say the things we do better is browser-side monitoring (session replay), event patterns/clustering, and we have first-party SDKs built on OpenTelemetry to make the setup a lot easier than vanilla OpenTelemetry.

I think Signoz has built a nice one-stop platform for observability, whereas we go one step further and focus on the developer experience to ensure anyone can fully leverage that observability data!

Thank you! I'm assuming this is in reference to our README? (Sorry I'm a _tad_ lacking in sleep)

If so, would you like to open a PR? I'm also happy to edit it myself but of course don't want to be stealing credit if you'd like to be attributed that way.

Ah got it, yeah a queue of some sort is definitely useful when scaling up to buffer pre-inserted data. This is something on the OSS side we've kept open to implementation. However it's something that is highly coupled with infra footprint and internal SLA guarantees the user wants to preserve. It can range anywhere from just rely on client-side retries to setting up a HA Kafka cluster early in the ingestion pipeline.

Similar to Elastic - I think a lot of architectures are available to choose on that side when users want to scale.

Will reach out to connect!

Thank you! We do support Winston (docs: https://www.hyperdx.io/docs/install/javascript#winston-trans...) and use it a lot internally. Let me know if you run into any issues with it (or have suggestions on how to make it more clear)

In fact this is actually how we develop locally - because even our local stack is comparatively noisy, we enable self-logging in HyperDX so our local logs/traces go to our own dev instance, and we can quickly trace a 500 that way. (Literally was doing this last night for a PR I'm working on).

Absolutely! It's pretty magical to go from a user report -> session replay -> exact API call being made and the backend error logs.

We dogfood a ton internally and (while obviously biased) we're always surprised how much faster we can pin point issues and connect alarms with bug reports.

Hope you give us a spin and feel free to hop on our discord or open an issue if you run into anything!

I'd genuinely would love to learn the OSS options we'd have available here, as we'd genuinely want to build a sustainable open source project and community, while preserving as many user freedoms as possible.

I think that HyperDX is a bit different from tools like Mongo, Redis or Hashicorp in that we're a vertically integrated product from SDKs/UIs to ingestion pipeline and DBs, which is opposite kind of offering from done by the above companies (which has made them more vulnerable to the kind of rug pull you mentioned)

We're trying to be permissive with freedoms granted to the user of our code, while still maintaining governance over the project to make it sustainable.

We don't want to be source-available, as that's pretty much the opposite of what we want to accomplish (and is why we consciously did not pick a license such as BSL/SSPL/etc.)

Our cloud features are closed source in a downstream repo - I think repos that have a very clear separation between OSS and closed are best - this also enforces that our OSS is always a fully-featured product that we develop on the OSS-only version day to day, and our cloud features are only a minor addition on top.

I've historically hit issues with repos that do an `ee` folder and blur the line between what is truly open source and self-hostable, vs need a license/cloud-only. I understand why they do that, but I hope we don't replicate that confusion ourselves :)

The absolute bare minimum I'd say is 2GB RAM, though in the README we do say 4GB and 2 cores for testing, obviously more if you're at scale and need performance.

For Syslog - it's something we're actually pretty close to because we already support Heroku's syslog based messages (though it's over HTTP), but largely need to test the otel Syslog receiver + parsing pipeline will translate as well as it should (PRs always welcome of course but it shouldn't be too far out from now ourselves :)). I'm curious are you using TLS/TCP syslog or plain TCP or UDP?

Here's my docker stats on a x64 linux VM where it's doing some minimal self-logging, I suspect the otel collector memory can be tuned down to bring the memory usage closer to 1GB, but this is the default out-of-the-box stats, and the miner can be turned off if log patterns isn't needed:

CONTAINER ID NAME CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O PIDS

439e3f426ca6 hdx-oss-miner 0.89% 167.2MiB / 7.771GiB 2.10% 3.25MB / 6.06MB 8.85MB / 0B 21

7dae9d72913d hdx-oss-task-check-alerts 0.03% 83.65MiB / 7.771GiB 1.05% 6.79MB / 9.54MB 147kB / 0B 11

5abd59211cd7 hdx-oss-app 0.00% 56.32MiB / 7.771GiB 0.71% 467kB / 551kB 6.23MB / 0B 11

90c0ef1634c7 hdx-oss-api 0.02% 93.71MiB / 7.771GiB 1.18% 13.2MB / 7.87MB 57.3kB / 0B 11

39737209c58f hdx-oss-hostmetrics 0.03% 72.27MiB / 7.771GiB 0.91% 3.83GB / 173MB 3.84MB / 0B 11

e13c9416c06e hdx-oss-ingestor 0.04% 23.11MiB / 7.771GiB 0.29% 73.2MB / 89.4MB 77.8kB / 0B 5

36d57eaac8b2 hdx-oss-otel-collector 0.33% 880MiB / 7.771GiB 11.06% 104MB / 68.9MB 1.24MB / 0B 11

78ac89d8e28d hdx-oss-aggregator 0.07% 88.08MiB / 7.771GiB 1.11% 141MB / 223MB 147kB / 0B 11

8a2de809efed hdx-oss-redis 0.19% 3.738MiB / 7.771GiB 0.05% 4.36MB / 76.5MB 8.19kB / 4.1kB 5

2f2eac07bedf hdx-oss-db 1.34% 75.62MiB / 7.771GiB 0.95% 105MB / 3.79GB 1.32MB / 246MB 56

032ae2b50b2f hdx-oss-ch-server 0.54% 128.7MiB / 7.771GiB 1.62% 194MB / 45MB 88.4MB / 65.5kB 316

I'd love to be using Cloudflare as our cloud provider, but it didn't seem to make a lot of sense for our use case.

We were concerned with some of the performance benchmarks we've seen with R2 in the past (though they've probably have improved), not to mention our compute options become a bit more limited to bandwidth alliance clouds otherwise we'll be eating network egress fees (which I do hate with a HUGE passion).

Though I can imagine if you're comfortable with one of the bandwidth alliance clouds already and can take a bit of a perf hit for search, R2 and Backblaze both can provide some cost savings depending on your workload.

I've tried liqe! I really wanted to love it - and I think it's amazing for the use case you've built it for, but I recall we ran into a few fatal issues (maybe it was supporting URLs or something as a property value?) and had to fork one of the `lucene` forks to get the grammar that we wanted.

Edit: happy to chat more about it as well if you're looking for more specific feedback - it's an area I've spent a decent amount of time on and would love to improve projects like liqe or others based on our experience if we can.

We use a mix of SSDs and S3 for storage depending on the workload - as you're right, merging on S3 is awful and we try to avoid it!

We're more than happy to have users self-host and deploy in a way that works with their SSO provider! Whether that's via SSO on Nginx or forking and adding SSO to Passport in their fork. Depending on the provider, it's likely very straight-forward to do.

We did explicitly choose MIT for the freedom of end users to deploy and modify the code how they want - and tried to open source pretty much everything that doesn't have a hard 3rd party dependency. We do touch a bit on how we think about the open core model as well in the README, and largely align with Gitlab's stewardship model [1] when it comes to paid vs OSS. In this case, a contribution to add SAML specifically to OSS will likely not be merged. It'd also introduce complexities with maintaining that alongside our cloud version that already includes a specific implementation of SAML.

[1] https://handbook.gitlab.com/handbook/company/stewardship/

Thank you, really appreciate the feedback and encouragement!

> It seems like clickhouse is obviously a big piece of the tech here, which is an obvious choice, but from my experience with high data rate ingest, especially logs, you can run into issues at larger scale. Is that something you expect to give options around in open source?

Scaling any system can be challenging - our experience so far is that Clickhouse is a fraction of the overhead of systems like Elasticsearch has previously demanded luckily. That being said, I think there's always going to be a combination of learnings we'd love to open source for operators that are self-hosting/managing Clickhouse, and tooling we use internally that is purpose-built for our specific setup and workloads.

> I saw what is in OSS vs cloud and I think it is a reasonable way to segment, especially multi-tenancy, but do you see the split always being more management/security features?

Our current release - we've open sourced the vast majority of our feature set, including I think some novel features like event patterns that typically are SaaS-only and that'll definitely be the way we want to continue to operate. Given the nature of observability - we feel comfortable continuing to keep pushing a fully-featured OSS version while having a monetizable SaaS that focuses on the fact that it's completely managed, rather than needing to gate heavily based on features.

> on OSS, I was surprised to see MIT license

We want to make observability accessible and we think AGPL will accomplish the opposite of that. While we need to make money at the end of the day - we believe that a well-positioned enterprise + cloud offering is better suited to pull in those that are willing to pay, rather than forcing it via a license. I also love the MIT license and use it whenever I can :)

> On that note, I am curious what your target persona and GTM plan is looking like?

I think for small teams, imo the options available are largely untantilizing, it ranges from narrow tools like Cloudwatch to enterprise-oriented tools like New Relic or Datadog. We're working hard to make it easier for those kinds of teams to adopt good monitoring and observability from day 1, without the traditional requirement of needing an observability expert or dedicated SRE to get it set up. (Admittedly, we still have a ways to improve today!) On the enterprise side, switching costs are definitely high, but most enterprises are highly decentralized in decision making, where I routinely hear F500s having a handful of observability tools in production at a given time! I'll say it's not as locked-in as it seems :)

The OpenTelemetry ecosystem is definitely still young depending on the language, but we have Ruby users onboard (typically using OpenTelemetry for the tracing portion, and piping logs via Heroku or something else via the regular Ruby logger).

Feel free to pop in on the Discord if you'd like to chat more/share your thoughts!

Definitely familiar with the compliance needs there - more than happy to chat further about BAAs and HIPAA compliance requirements with you guys. Always love partnering with others in the OSS space :)

This one is a fun one that I've spent too many nights on - we're largely similar to Google-style search syntax (bare terms, "OR" "AND" logical operators, and property:value kind of search).

We include a "query explainer" - which translates the parsed query AST into something more human readable under the search bar, hopefully giving good feedback to the user on whether we're understand their query or not. Though there's lots of room to improve here!

I designed our landing page - and I definitely took heavy inspiration from Linear. As an engineer, creating novel beautiful design's isn't first-nature to me, but I know how critical it can be to make a clean/impactful landing page so I try to take some elements from the best.

Some other landing pages I loved and had along side while designing ours were Vercel, Resend, and WorkOS :)

Let's say my React profiler tab gets lots of love - though if you find anything sluggish, let us know and I'd love to fix it. The last thing I want HyperDX to feel like is Jira sluggishness.

We should have both! If you hover over a property value, a magnify/plus icon come up to allow you to search on that property value (no manual facets required) - and our traces all come with delicious flame graphs :) Let me know if you were thinking of something different.

One other thing I think you'd love if you're coming from Datadog is that you're able to full text search on structured logs as well, so even if the value you're looking for lives in a property, it's still full text searchable (this is a huge pain we hear from other Datadog users)

If there's anything you love/hate about Datadog - would love to learn more!

Thank you! Absolutely agree on Datadog/New Relic DX, I think the funny thing we learned is that most customers of theirs mention how few developers on their team actually comfortably engage with either New Relic or Datadog, and most of the time end up relying on someone to help get the data they need!

Definitely striving to be the opposite of that - and would love to hear how it goes and any place we can improve!

We've found that across most platforms, Otel is more of a starting point to building good instrumentation libraries. We've been building on top of Otel/Splunk's browser SDK implementation for our package [1] which does roll in session replay, and also adding on better exception tracking, etc. None of which remotely comes out of the box unfortunately.

Being able to connect frontend sessions -> backend trace/logs has been a huge DX change though imo.

[1] https://www.hyperdx.io/blog/browser-based-distributed-tracin...