chasers

I assumed it did because I would think Postgres caches the query plan by some statement hash not just by name.

Anyways, I'm probably wrong here but it looks like I'm about to figure this out now :D

It is very BEAMy because we love the BEAM. All connections are supervised :D

Might be a little too close to "supervisor" though.

> To what extent are these metrics distorted by use of a Supavisor connection pooler? How could Supavisor compensate for that?

You mean if you have an Elixir app and using Supavisor for connection pooling? They would be affected as much as they would be if you used PgBouncer.

For the metrics Supavisor is exposing, it has those Ecto metrics for the interactions with the management database.

Supavisor doesn't use any libs for handling the wire protocol. It handles the wire protocol directly. We are exposing some similar ones: https://github.com/supabase/supavisor/wiki/Metrics

Thanks!

You could bring it in as an application but you already get pooling with Ecto via Postgrex and DBConnection.

If you're using Elixir there's really no need for separate connection pool until you have LOTS of nodes.

Yeah what you're looking for is "unnamed prepared statements". If your client supports this then you can use PgBouncer and get the throughput benefits.

Maybe binary_parameters is a different thing. Need to look into it more but this is a good one to add to the docs. And maybe officially support named prepared statements natively if lots of clients don't support unnamed prepared statements but definitely seems like we'd have to do much more accounting.

Unknown at the moment but we're definitely open to some prior art if there's an implementation you appreciate.

Naively, POSTing a query string and params would go a long ways I think.

Can your client use the `binary_parameters` param?

e.g. https://blog.bullgare.com/2019/06/pgbouncer-and-prepared-sta...

Elixir's Ecto can (and that Go example) so if you use Elixir (or Go) you can use prepared statements with PgBouncer (or Supavisor!) in transaction mode.

We discussed this. Don't remember exactly but @abc3erl had a lot of experience with poolboy.

We made sure to do local benchmarks along the way and there wasn't a meaningful difference.

Really the discussion was around doing as little message passing as possible to not copy data.

See:

https://github.com/supabase/supavisor/pull/4

https://github.com/supabase/supavisor/issues/7

So there are definite gains there but if we want to do things across a cluster we're going to be copying data. The decision at the time was the benefits of clustering outweighed the perf gains considering we were basically already as fast as PgBouncer. This could change.

Ha! Wait but why Google Cloud SQL proxy too?

If you are interested in connecting over this it would be great to connect… chase@supabase.com.

We want to do load balancing too. Makes so much sense to do load balancing in the proxy.

Nice! Yeah really our first go at it so I’m sure we have room to improve.

PgBouncer was on the same instance I’m assuming?

Binary pattern matching is great for working with protocols.

Erlang process monitoring makes it very easy to handle client disconnects.

And Elixir makes using Erlang a bit more approachable.

Contributor here. Happy to answer any questions.

An interesting bit... the syn library helps us guarantee there is only one connection pool alive on the cluster at a time and gives us callbacks to resolve the conflict if two get spun up.

https://github.com/ostinelli/syn

Pattern matching and structs get you 90% of the way there imo. And Elixir started emitting warnings if you try to access a field in a struct and it doesn't exist.

Yeah the database ultimately guarantees that. This lets us distribute the fact that a connection exists (in the form of a PID) so that we don't have to go back to the db on every connection request. This helps quite a bit in the case of a lot of connections coming in very quickly, especially when those connections are geographically far from the database or the database is under load from some other system.

If no connection exists yet, you're right, two connections could try to establish themselves at the same time. In which case, whoever gets to the db first wins. But then the loser is still connected to PubSub, so they'll now start getting changes just the same.

Yeah I plan to. Have some things to prep next week for ElixirConf but am going to get back to it after that.

Pricing is still tbd unfortunately. Currently every project gets these features with some limitations:

- 25 concurrent channels

- 25 channel joins per second

- 200 concurrent connections

- 100 broadcast messages per second avg over 60 seconds (not including db changes) (aka 6,000 per minute rolling)

I think these are probably pretty close to where we'll land for free projects. They can be changed per project so if you (or anyone) needs a limit increased in the short term we can do that.

There is actually a case study on elixir-lang.org of exactly this!

https://elixir-lang.org/blog/2021/01/13/orchestrating-comput...

Yeah this is one of the reasons for these new features. So you don't have to send things like cursor positions, currently online, typing, etc through your database.

Definitely feels like the beam was built for this kind of work.

Thanks Jose! I picked up Elixir as my first language and have really enjoyed it.

> I think the interesting part is the overlap between observability metrics for operations and expanding in to BI metrics with the same tools.

I built Logflare so you could get your structured logs into BigQuery directly, so now you this :)

Hopefully we'll have a Vector sink soon, but until then I think they support POSTing batches to HTTP endpoints so you can do that and we'll take any JSON and go straight to BQ with it after migrating your schema for you (automatically based on the incoming payload shape).

Only three metadata fields or can you have more?

So event sourcing?

If anyone else was curious...

"Our tracing pipeline has been in production for over a year now and we trace 1% of all requests from our clients. For some low volume services, we trace 100%. Our current pipeline processes ~310M traces/day and about ~8.5B spans per day, producing around 2Tb of trace data every day."

> Processing/streaming logs to get metrics is a terrible waste of time, energy and money.

Only if you're using Elastic.

Kind of defeats the purpose. If you can bail you don't have to figure it out.

This is awesome! Sure the data is not perfect but it's what we've for and the r naught is exactly what we need to be monitoring.

Google Trends scraper.

Elixir :)

It sounds like they were maybe pattern matching to strictly. If the value is always a string just save the empty string and try to discover that service. If it doesn't exist, oh well, it doesn't crash the process.