Minimal, allocation-free OpenMetrics implementation for no-std/embedded Rust | Hacker News Reader

Minimal, allocation-free OpenMetrics implementation for no-std/embedded Rust

github.com

170 pointsingve24 comments

hi, i wrote this thing!

if you’re wondering why it’s weird, half-finished, or under-documented, it’s because i wrote it in a couple hours to scratch my own itch, and really didn’t expect to be at the top of hackernews today! if this is something that other people are actually interested in using, i’d be happy to clean it up a bit and add some of the missing stuff…

Thanks for doing that! I think there is demand for minimal metrics services for Prometheus for very constrained or expensive environments.

I am currently looking into getting container-level metrics into Fargate containers.

I think the best way forward, if you want to keep working on it would be to provide good documentation on how to use it. As it reads from the README now, you need to set the amount of labels you will have before compiling? I am not a Rust user so basically have no idea how to test that thing.

For a feature request, I think histograms would be best.

Thanks again and good luck!

How do people tend to get metrics off of embedded devices?

I’m working on autometrics (https://github.com/autometrics-dev/autometrics-rs) and people asked whether it could be used in embedded contexts but I wasn’t sure how you’d hook up the device to something like Prometheus.

Prometheus push gateway maybe? I guess it depends on limitations like battery life, bandwidth and network access (maybe it's behind a NAT or something)

The Pixel phones have an embedded environment hooked up to the normal ARM/Linux SoC. So you pipe from embedded to Linux and then Linux relays up to the cloud.

in my use case (a hobby project; https://github.com/hawkw/eclss), the device and Prometheus are on the same LAN and i have Prometheus set up to discover scrape targets using multicast DNS. this is…probably not what you’d do if you were shipping a consumer project, i think, but it’s a nice setup for a home network.

Ah, that is indeed a cool setup for a device on your home network :)

For embedded systems probably just... allow users to read the metrics easily and let them implement what they need using the method of pushing they use. use-cases will probably be different enough that they'd need to customize it anyway.

Most would probably end up as pushing metrics via... something (MQTT?) to gateway that will push it to their metrics system.

Pulling/polling metrics honestly is pretty bad idea most of the time. It makes it easy to implement ("just* provide a web page with metrics) and allows poller to control intervals (which is only a problem if your configuration management isn't up to snuff), but complicates a lot, now you need to setup whole endpoint discovery instead of just saying "hey, apps, here is URL to push your metrics to, have fun".

We for long (10+ year) have system based on collectd and it's line protocol, and while it have some drawbacks (no tags, just rigid structure with host->plugin->plugin_instance->type_type instance, works well most of the time but 100% of the time), push nature is very light on resources (it's just an UDP packet full of metrics, no need to setup whole TCP connection, doesn't even need bidirectional firewall)

Polling has several benefits over pushing: a misbehaving endpoint can't knock out the whole system by having too much data, but at worse will just not get polled. Similarly load shedding is pretty simple.

In my experience the discovery and configuration is needed anyway to do something with the metrics.

Small embedded devices tend have a serial port of some sort and some ad-hoc protocol or mechanism to trigger metric output. A prime example of this that comes to mind is the P1 port on some of the European electricity meters. This user-accessible port outputs some metrics every so often. Specification for the curious is here: https://www.netbeheernederland.nl/_upload/Files/Slimme_meter...

ARM chips intended for embedded uses also have a single wire output peripheral for tracing and similar purposes that could be used for this.

Memfault is another option.

We had a need for telemetry recently so I looked at OpenMetrics libraries.

Basically they didn't allow sufficient control for low-latency applications so we just rolled our own approach that writes samples to InfluxDB.

OT but still sooo weird to me the text exposition format is the only format.

for embedded devices (like this lib targets), having the now-removed protobuf exposition format seems like such an easy win. less memory & less work & less bandwidth.

It was added back to Prometheus.

https://github.com/prometheus/docs/blob/main/content/docs/in...

I think the reason for that is that Protobuf doesn’t provide streaming parsers (yet). As the Protobuf based protocol stored all samples in a single message, this meant that both the client and server had to allocate lots of memory when sending/receiving metrics.

With the text based protocol this was never an issue, as the process that does the scraping can ingest metrics line by line.

Why do you care? Even if you have thousands of metrics, this is still relatively little data.

This also presumably runs on an auxiliary thread, and uses a network interface not tied to production services.

Not the same type of metrics, and not Rust, but C++ - but there is also https://perfetto.dev/ which has custom (protobuf-based) protocol, but the generated code is not with the standard protoc, but custom one https://perfetto.dev/docs/design-docs/protozero - with some limitations (last time I checked you can't modify fields, just add values of sorts - which makes sense in the embedded case)

How do you make things allocation free? and until how much degree you call something allocation free?

> you can't (or don't want to) allocate memory dynamically. this crate is intended to allow all metrics storage to be declared in statics, for use in embedded systems and other no-std use-cases.

Basically, the library just transforms over a statically assigned buffer.

In a lot of languages, this means you lose out on some language features, but I believe features like pattern matching are alloc free in rust.

More accurately for a lot of languages you can’t make them allocation free. Rust put all of the parts of the standard library that allocate in one crate, called alloc, and that crate is optional (but included in std, so opt-out). Platforms that don’t have an allocator don’t have the alloc crate. So you are forced to select dependencies like this one that will compile without it.

Language features like pattern matching are not affected.This is about standard library features. You don’t get Vec or String without the alloc crate.

Takes some knowledge of the std lib of whatever language you're using (easy with C and Rust, harder with C++) to know what calls will try to allocate memory. But another method is to use a static buffer of bytes designated in the link table as your "heap" and have tasks only allocate when they start, and ensure they do not free that memory. Algorithmically, allocation is the easy part, reusing freed blocks is more difficult. So if your embedded allocator doesn't free and faults at OOM, if you allocate only at the start of a program, you can still be more confident in memory safety

The problem with allocating is not the algorithm complexity, it's the fact it may need to allocate new pages from the operating system, which is a somewhat expensive operation.

If you use a global allocator, when this happens is entirely non-deterministic. If you use a local iterator, at least you control when that happens, and have guarantees on the asymptotical behaviour.

^I had the same question, thank you for asking! To extend, OP can you link a few code lines in the project which shows this allocation free magic?

Hey I was the first rust metrics author. Glad to see this project and it’s a good step forward

Reply on news.ycombinator.com