Comment by ahachete | Hacker News Reader

Comment by ahachete | Hacker News Reader

IMO transparent proxies for observability are not the best pattern. And I speak from experience, we developed the Postgres plugin for Envoy [1], [2] and we use it in StackGres [3], among others, for this very same reason, observability.

There's two main problems with said proxies:

* Latency. Yes, yes, yes, they add "microseconds" vs "milliseconds for queries", and that's true, but just part of the story. There's an extra hop. There's two extra sets of TCP layers being traversed. If the hop is local (say a sidecar, as we do in StackGres) it adds complexity in its deployment and management (something we solved by automation, but was an extra problem to solve) and consumes resources. If it's a network hop, then adds milliseconds, and not microseconds.

* Performance. It's not that hard to write a functioning PG wire proxy (it's not trivial either). But it is extremely hard to make it perform well under high load scenarios. Most of the proxies I have seen crack down under moderate to high performance.

What's the solution then? The Postgres extension model to capture the metrics (we also experimented with eBPF, but it causes too many kernel-user space context switches when you can do the same in an extension without them), and a small sidecar to push the metrics out via a standardized protocol like OTEL.

[1]: https://www.envoyproxy.io/docs/envoy/latest/configuration/li...

[2]: https://www.cncf.io/blog/2020/08/13/envoy-1-15-introduces-a-...

[3]: https://stackgres.io

Edit: formatting

I get what you're saying about a proxy like this, latency & performance would suffer, however minor, and in production DB land this really matters.

I've just not sure it is much of a slight on such proxies.

You don't need to run this always inline in production to get amazingly useful results. Yes, there are lots of production insight solutions out there, but lots of modern stacks can be complex enough that just getting a quick handle on how the page you're debugging talks to your DBs can be incredibly useful, which is where I love the idea of a solution like this.

Sure, it is mytop / pgtop, but trying to offering it at a different layer & with a modern interface. Seems useful to me.

> The Postgres extension model to capture the metrics (we also experimented with eBPF, but it causes too many kernel-user space context switches when you can do the same in an extension without them), and a small sidecar to push the metrics out via a standardized protocol like OTEL.

The extension model is great, but it doesn't work with existing postgres providers (RDS, Aurora, etc.). Unless one such extension becomes standard enough that all providers will support it. That would be ideal, IMO.

To be clear, I don't mean pg_stat_statements, that is standard enough, but an extension that pushes the actual queries in real-time.

> If it's a network hop, then adds milliseconds, and not microseconds.

Are you talking about connection establishing time or for query delay? I think it should normally be under a millisecond for the later.

> The extension model is great, but it doesn't work with existing postgres providers (RDS, Aurora, etc.). Unless one such extension becomes standard enough that all providers will support it. That would be ideal, IMO.

That's true, but that's a problem of the PGaaS providers for them to fix by providing the best functionality available. I'm planning on following this route in a pure OSS basis.

> Are you talking about connection establishing time or for query delay? I think it should normally be under a millisecond for the later.

Network trip. If the proxy is not co-located with the database but rather a network hop away, that's usually adding at least 1ms there, could be more.

Reply on news.ycombinator.com