We had to push metrics we scrape via Prometheus into DataDog (coincidence that they acquired this) and do a custom transform to map to a set of custom metrics.
Very straightforward in how it runs and the helm chart had all the right things in there
I'm excited for these front-end telemetry routers to keep going. Really hoping Vector can co-evolve with and grow with the rest of the telemetry ecosystem. Otel itself has really started in on the next front with OpAMP, Open Agent Management Protocol, to allow online reconfiguration. I'd love to know more about Vector's online management... quick scan seems to say it's rewriting your JSON config & doing a SIGHUP. https://opentelemetry.io/docs/specs/opamp/
Vectors configurability & fast-and-slim promise looks amazing. Everyone would be so much better off if it can grow to interop well with otel. Really hoping here.
Don’t get me wrong, I want to use OTEL, but it’s a struggle. In the meantime, I’ve still got normal apps and libraries outputting normal logs and normal prom metrics, so I’ve got to stick with that.
Or are you thinking more on the UI/analysis/collection side?
Oh you need a collector? But maybe you don’t - because some libs will push it? Ok so we got that setup, but now half the traces don’t turn up? Or they do, but they’re missing the ids to link them together? I’ve got 30m traces from an AWS lib we use, but none of ours? Oh also our logs don’t come across? Because logs require some different handling or something and some intermediary didn’t support them yet? Grafana seemed to support some things, and not others. To say nothing of the absolute plethora of config options available on the collectors and exporters: there’s like 3 or 4 different ways to define sampling and filtering, in a different layer each and they all appear to cross interact, so you can accidentally choose configs that prevent you from getting data with no indication of where it’s gone missing.
I’m keen for it to all shake down a little bit, because I’d love to be able to just bang #[instrument] on all our functions, and derive logs and metric from trace data, but seems things are a while off that yet.