Oh yeah Rust... definitely a bit more sharp edges there which is frustrating (had a friend tell me the same recently).
674 karma · joined July 8, 2014
(We've been working on some Otel-based SDKs to try to make it easier to onboard [1] as an example, though curious where the general sentiment is)
Supposedly Google has spent $1.1bn between 2009 and 2015 [0], and Waymo has raised $5.5bn in outside funding, supposedly last valuing it at around $30bn. Cruise is apparently on track to burn ~$2bn/yr. Getting reliable figures on actual dollars invested to date on projects is much harder unfortunately. However, it seems like overall it's in the range of billions and likely less than $10bn for any individual project. [1]
As a few benchmarks, I was curious about the cost of the California HSR, Caltrain electrification, and the recent SFMTA Central Subway project.
The SFMTA Central Subway project was a 1.7 mile extension of an existing T line, and cost $1.95bn.
The California HSR's initial 171 mile segment is estimated to cost $35 billion with the final system estimated to be well north of $100bn.
The Caltrain electrification electrified 51 miles of the Caltrain network and introduces new EMU trains at the cost of $2.44bn.
From a bay area resident/layman perspective, the R&D cost for any self driving car project doesn't seem to be out of proportion compared to public transit. Even if we took the information's estimate of the whole industry burning $16bn so far, that'd only buy you half of the initial segment of the Cailfornia HSR, which doesn't make it seem like self-driving car research is that outlandish. In fact I wouldn't be surprised if full autonomous vehicles land before the HSR fulfills its full mission of LA <> SF in under 3 hours.
Bloated local projects like the central subway make self driving car investments a steal, with 1.7 miles of subway in SF costing the same ballpark as a semi-mature self driving car program.
[0] https://spectrum.ieee.org/google-has-spent-over-11-billion-o... [1] https://www.theinformation.com/articles/money-pit-self-drivi...
The automatic instrumentation part is for tracing, which arguably is a _huge_ value add on top of logging, but is indeed more complicated to get setup.
Unfortunately I think OpenTelemetry doesn't do a great job in expressing the different parts of the project and how they all fit together (though this is the status quo in observability in general). I think third party vendor documentation tend to do a better job in explaining how to get started with it without needing to piece together the exporters, batchers, context managers, etc.
Disclaimer: I'm building hyperdx, an otel-based OSS observability tool, so I'm pretty biased in being pro-OpenTelemetry :)
If this was us [1] (an OSS Datadog alternative posted last week :D ), we do have quite a bit (11, 2 are optional) but we're working on bringing it down.
We currently split our ingestion pipeline into 3 independently scalable bits, but we can probably bring it down into 1 for any small-scale deployment. Otherwise we do have the standard need for a cache (redis), db (mongo), main storage (clickhouse), and then the standard API server + frontend, and a separate task to run alerts.
There's likely a few more things we can merge together, but it comes at the expense of making it unscalable to workloads typical to a company and divergence in the code base, which overall doesn't seem like the right tradeoff.
Imo the real concern as someone that personally owns too many tiny VM instances is resource footprint - which can be tuned down to ~1GB of memory for us (depending on server load).
Feel free to open an issue too if you think there's something we can adjust there as well.
Otherwise you can use the OpenTelemetry collector[2] to do the same[3].
[1] https://www.hyperdx.io/docs/install/docker
Generally pretty good way to approach research papers in a new field I recall being taught, there's always some sort of "landscape overview" paper being published that can help distill down the SoTA and you can just follow the references.
Clickhouse gives users much greater flexibility in tradeoffs than either a time-series or inverted-index based store could offer (along with S3 support). There's nothing like a system that can balance high performance AND (usable) high cardinality.
[1] https://github.com/hyperdxio/hyperdx
disclaimer (in case anyone just skimmed): I'm one of the authors of HyperDX
The technical details are best explained by the authors of the original paper [1]. We weren't smart enough to come up with it on our own and can't take credit for that haha
As for DPAs - yes!
I'm thinking if someone self-hosts a complex OSS app like Dittofeed (or even HyperDX ;) ) - would it be beneficial to allow admins to enable email onboarding flows like they get in the SaaS side of things, so end-users can be drip-fed product docs appropriately (and even customize it themselves if they're a big enough enterprise)? This can probably help onboarding/adoption internally if done well.
Either way - I'm really excited to try this out on our cloud side of things to help our new users learn our platform faster :)
We're thinking of being able to allow users to create a custom webhook alert so you can get full flexibility on what vendor you use and how the alert should be crafted, would love to hear your feedback there, though may need to ship some stopgap solutions depending on demand!
Of course if your local/prod run completely different and require different instrumentation, that might be trickier.
I'm wondering if you had a specific use case in mind? Happy to dive more into how it should be done (feel free to join on Discord too if you'd like to chat there)
I'd also add that I don't think all services trend their price upwards. AWS has historically lowered prices on services and continue to offer new service-tiers with lower prices (S3 tiering as an example). As the tech matures and costs fall for our service as well, it'd be surprising if we don't do the same.
For now we're trying to make the base price cheap enough where those kinds of considerations don't need to be top of mind today and a policy that can be forgiving when it occasionally happens, but certainly as we continue to scale and grow, we'll need to put in proper controls to allow users to define what should happen if events are spiking unexpectedly (how to shed events via sampling, what needs to be explicitly perserved for compliance reasons, when to notify, etc.)
I do like Sentry's auto-sampling algorithm which is a really neat way to solve that issue.
For automatic logging - I think you'd enjoy OpenTelemetry's automatic tracing implementation, it helps pull out standard telemetry from things like your Redis requests and correlate them with trace IDs so you can tie everything together from the moment your server starts accepting the HTTP request to the Redis and DB requests and what was sent in each request (without needing to do it manually)
For capturing HTTP req/res - we actually have a few options depending on the language (ex. we do this for Python and Node.js) to enable more advanced network capture (so you can actually get the full req/res information, or whatever subset you're interested in actually storing)! It's actually been asked by a number of teams to make it easier to debug tricky API issues they're running into.
Proper end-to-end logging definitely makes it easier to find the right clue among a sea of logs, hopefully we make it easier to get there!
HyperDX:
- Can collect all server logs (to help debug issues even if an exception isn't thrown)
- We can collect server metrics as well (CPU, memory, etc.)
- We accept OpenTelemetry for all your data (logs, metrics, traces) - meaning you only need to instrument once and choose to switch vendors at any time if you'd like without re-instrumenting.
- We can visualize arbitrary data (what's the response time of endpoint X, how many users did action Y, how many times do users hit endpoint X grouped by user id?) - Sentry is a lot more limited in what it can visualize (mainly because it collects more limited amounts of data).
Sentry:
- Great for exception capture, it tries to capture any exception and match them with sourcemap properly so you can get to the right line of code where the issue occurred. We don't have proper sourcemap support yet - so our stack traces point to minified file locations currently.
- Gives you a "inbox" view of all your exceptions so you can see which ones are firing currently, though you can do something similar in HyperDX (error logs, log patterns, etc.) theirs is more opinionated to be email-style inbox, whereas our is more about searching errors.
- Link your exceptions to your project tracker, so you can create Jira, Linear, etc. tickets directly from exceptions in Sentry.
I don't think it's an either/or kind of situation - we have many users that use both because we cover slightly different areas today. In the future we will be working towards accepting exception instrumentation as well, to cover some of our shortfalls when it comes to Sentry v HyperDX (since one common workflow is trying to correlate your Sentry exception to the HyperDX traces and logs).
Hope that gives you an idea! Happy to chat more on our Discord if you'd like as well.
A few of the weaknesses of S3 are:
1. API calls are expensive, while storage in S3 is cheap, writing/reading into it is expensive. Using only S3 for storage will incur lots of API calls as Clickhouse will work on merging objects together (which require downloading the files again from S3 and uploading a merged part) continuously in the background. And searching on recent data on S3 can incur high costs as well, if you're constantly needing to do so (ex. alert rules)
2. Latency and bandwidth of S3 are limited, SSDs are an order of magnitude faster to respond to IO requests, and also on-device SSDs typically have higher bandwidth available. This typically is a bottleneck for reads, but typically not a concern for writes. This can be mitigated by scaling out network-optimized instances, but is just another thing to keep in mind.
3. We've seen some weird behavior on skip indices that can negatively impact performance in S3 specifically, but haven't been able to identify exactly why yet. I don't recall if that's the only weirdness we see happen in S3, but it's one that sticks out right now.
Depending on your scale and latency requirements - writing directly to S3 or a simple layered disk + S3 strategy might work well for your case. Though we've found scaling S3 to work at the latencies/scales our customers typically ask for require a bit of work (as with scaling any infra tool for production workloads).
If anyone wants to chat more about this - I've kicked off an issue [1] to gather interest and everyone's feedback.
Otel tracing works and should be pretty bulletproof - metrics is still early so you might see some weirdness (we'll need to update the remaining work we've identified in GH issues)
You can 100% build tabular views based on JSON logs, we auto-parse JSON logs and you can customize the search table layout to include custom properties in the results table.
Let us know if we fulfill this need - we at least do this ourselves so I feel pretty confident it should work in your use case! If there's anything missing - feel free to ping us on Discord or open an issue, we'd likely benefit from any improvement ideas ourselves while we're building HyperDX :)
Edit: Oh I also talk a bit about this in another comment below https://news.ycombinator.com/item?id=37561358
Grafana/Prom/Loki is an awesome stack - overall I'd say that we try to correlate more signals in one place (your logs <> traces <> session replay), and we also take an approach to go more dev-friendly to query instead of going the PromQL/LogQL route.
It's a stack I really wanted to love myself as well but I've personally ran into a few issues when using it:
Loki is a handful to get right, you have to think about your labels, they can't be high-cardinality (ex. IDs), the search is really slow if it's not a label, and the syntax is complex because it's derived from PromQL which I don't think is a good fit for logs. This means an engineer on your team can't just jump in and start typing keywords to match on, nor can they just log out logs and know they can quickly find it again in prod. Engineers need to filter logs by a label first and then wait for a regex to run if they want to do full-text search.
Prometheus is pretty good, my only qualm is again the approachability of PromQL - it's rare to see an engineer that isn't fluent with time-series/metric systems to be able to pick up all the concepts very quickly. This means that metrics access is largely limited to premade dashboards or a certain set of engineers that know the Prometheus setup really well.
Grafana has definitely set the standard for OSS metrics, but I personally haven't had a lot of success using their tools outside of metrics, though ymmv and it's all about the tradeoffs you're looking for in an observability tool.