BenjieGillam

Yep; we have a dedicated listener client per pool (and normally something like 1-20 pools, each with 1-100 workers); the workers themselves don’t listen. We also notify on every job or batch job insert, though we would love to rate limit this so we notify at most once per 2 milliseconds.

Completely agree; the issue I see with people who try putting logic in the DB ultimately to reject it is they try and bring their procedural programming paradigms (e.g. “for” loops and “if” statements) into the DB and that’s incredibly inefficient, requiring row-by-row looped fetches and ultimately causing their systems to grind to a halt and them to blame the concept of putting logic in the DB. If instead they were to embrace the declarative nature of the database and used efficient patterns such as statement triggers rather than row triggers, common table expressions (CTEs) rather than row-by-row looped processing, set-returning functions and joins rather than row-by-row function calls, etc then they would find that their code is both more expressive and often thousands of times more performant. As you say, embracing this model means less data transferred between cluster and client which can result in reduced load across the entire system without increasing complexity or having to worry about cache invalidation issues. And time spent learning databases is knowledge that will still be useful in a decade or two, it’s well worth investing into.

Yes, currently Node is the runtime but we could bundle that up into a binary blob if that would help; one thing to download rather than installing Node and all its dependencies?

A UI is a common request, something I’ve been considering investing effort into. I don’t think we’ll ever have one in the core package, but probably as a separate package/plugin (even a third party one); we’ve been thinking more about the events and APIs such a system would need and making these available, and adding a plugin system to enable tighter integration.

Could you expand on what’s missing in the documentation? That’s been a focus recently (as you may have noticed with the new expanded docusaurus site linked previously rather than just a README), but documentation can always be improved.

Not sure if you saw it but Graphile Worker supports jobs written in arbitrary languages so long as your OS can execute them: https://worker.graphile.org/docs/tasks#loading-executable-fi...

Would be interested to know what features you feel it’s lacking.

Its an optional command line utility that you may use with PostGraphile which does things like printing out your configuration in a pretty format and using TypeScript to figure out what options are available to you based on the plugins you are using. It is 100% non-essential because all the options are documented in each of the plugins (and also you can use TypeScript auto-complete in your editor), and you can just console.dir() your configuration. You can read about it here: https://postgraphile.org/postgraphile/next/config#viewing-th...

You can use absolutely any migration framework you like with PostGraphile, it’s completely unopinionated about that.

You have to get used to mentioning it in every available avenue (readmes, docs, CLI greeting message, any web UIs, release notes, issue templates, etc), and also be happy with a really slow ramp-up time, but yes, it’s possible. Source: I’m Benjie on GitHub if you want to check out my sponsors profile.

Graphile Worker maintainer here; keep in mind that postgres is not the ideal location for a job queue, so you’re going to be limited ultimately by postgres’ capabilities. I’ve seen Worker max out at around 10k jobs/second but very much YMMV - you should benchmark it for your expected use case. Personally I’d move to a dedicated job queue if I started having an average of more than 1-2k jobs per second. The majority of systems never hit anywhere near this (we very much cater to the “long tail” of job queue needs).

Regarding the horizontal scalability; that relates to if you have heavy tasks (tasks that take a second or more to execute) - you can use more instances to get higher throughput.

Hope this helps!

> Postgraphile, et al. all seem to suggest that you should stand up a separate service for this.

PostGraphile maintainer here; with the exception of recommending job queues for work that can/should be completed asynchronously (which is a good idea no matter what server you use) I do not recommended setting up a separate service for this kind of thing. PostGraphile is highly extensible, you can implement most things in JS (or TypeScript) natively in Node.

That's not true; you can use row level security (RLS) to control access (both reading and writing) on a per-row basis. You can think of it as similar to an implicit "where" clause that automatically gets added to all requests.

RLS: https://www.postgresql.org/docs/current/ddl-rowsecurity.html

Most PostGraphile projects have two or three PostgreSQL roles total, and that can support millions of application users. This is such a common misconception that we created an infosheet about it: https://learn.graphile.org/docs/PostgreSQL_Row_Level_Securit...

Or use an API layer such as GraphQL that doesn’t require versioning for such a minor change since additive changes (or deprecations) to the API do not affect existing queries - each query states exactly what it needs.

Please note you can add custom handlers in PostGraphile using JS as well as SQL; we have an extensive plugin API, but if you just want to use SDL and resolvers we’ve a plugin generator for that: https://www.graphile.org/postgraphile/make-extend-schema-plu...

Apparently you can’t reply with just emoji here, so: _high five emoji_

Graphile Starter-based stack (Node, PostGraphile, Next.js, Graphile Worker) running on Heroku with Amazon RDS Postgres. Virtually no server maintenance needed: just push the code to GitHub, check it works on staging, press the promote to production button, job done. (I’m the Graphile maintainer.)

PostGraphile is designed for a situation where Postgres is your main data store, and other things are ancilliary. You can add in the other things with traditional GraphQL resolvers via the plugins/etc that you've mentioned; but if your stack is already quite complex with a lot of data stores that interact in interesting ways (i.e. not just driven by postgres and then fetching additional things from other stores as needed; but instead getting some from one, feeding it to another, feeding it back to the first, etc) you might be better off rolling your own GraphQL API that can handle your complex requirements.

If you like the plugin system, lookahead, etc then you could dive straight into Graphile Engine (which is agnostic to data store(s)) and use that to build your GraphQL API. If autogeneration and applying wide-ranging transforms appeals to you, that might be an interesting approach, but otherwise that system is not sufficiently optimised (yet) for one-off APIs currently (and is under-documented), so you might have a better time just writing your own types and resolvers with Apollo Server or similar.

Thanks! Makes sense; best of luck with your future projects!

Out of interest (as the PostGraphile maintainer) did you look into https://www.graphile.org/postgraphile/make-extend-schema-plu... for extending the PostGraphile schema to do whatever you need, or were you specifically looking to implement the extensions in Go?

[PostGraphile author here, and I wrote that page of documentation.]

Firstly, GraphQL does not allow for infinite recursion; it is literally not possible to do infinite recursion in GraphQL; the GraphQL spec even has a section on this: https://spec.graphql.org/draft/#sec-Fragment-spreads-must-no...

Secondly, it's extremely easy to add a GraphQL validation rule that limits the depth of queries; here's an example of one where it takes just a single line of code: https://github.com/stems/graphql-depth-limit . This isn't included by default because there are plenty of solutions you're free to choose between, many of which are open source, depending on your project's needs. For most GraphQL APIs, persisted queries/persisted operations is the tool of choice, and is what Facebook have used internally since before GraphQL was open sourced in 2015. (Unlike what you state, this does not turn your API into a "REST API," it acts as an optimisation on the network layer and once configured is virtually invisible to client and server.)

I’m the maintainer of PostGraphile and I also don’t approve of “just expose your database to the world and let your frontend just get whatever data it needs.” PostGraphile is a tool to help you to rapidly build performant and secure GraphQL APIs that follow many of the GraphQL best practices out of the box. It is not a “run and done” solution, you do have to put a little effort in to build a decent API but we give you the tools to achieve this in a fraction of the time of rolling your own, and probably with better performance also. Our extensive documentation is peppered with advice on how to craft your API, not to expose more than you need, not to expose complex filters (instead adding specific filters the frontend needs), how to include/exclude things, how to add in additional fields, types, remote content, etc. Our users are generally very happy, we’re just not very good at marketing ;)

<checks the income from the pro plugin versus that from sponsorship> I can definitely confirm that we're significantly more personal/community driven than commercial.

No worries, I think judging things we're unfamiliar with is very human - it's necessary to protect our own time because we can't afford to research _everything_. We just need to be careful to not share these quick judgements as if they are well researched opinions.

> I just feel that writing business logic should not be something you have to do in a plugin of some tools or with knowledge of graphql (resolvers)

Business logic doesn't need to be written in the resolvers themselves (this is not generally good practice); instead your resolvers call out to your business logic layer as is the GraphQL way. For PostGraphile, in general the "business logic layer" is in the database itself (e.g. PostgreSQL functions, etc) which can be reused directly by other components of your stack (e.g. a REST API), but it doesn't have to be.

Is your issue more with how GraphQL in general works (resolvers calling out to the business logic layer), or do you feel that PostGraphile is adding layers of indirection due to an issue with the term "plugin"? If we called them "modules" would you be happier? Here's an example "plugin" that adds a new field to our GraphQL API, where the business logic is contained in a fictitious npm library and we just defer to that in the resolver - this is a common way of writing GraphQL schemas: defining the types/fields via SDL and the accompanying resolver functions: https://gist.github.com/benjie/68f55fa1bcc07e1cd7a8f49d423f1... . In PostGraphile we handle this through plugins because it allows you to mix and match (should you wish) different ways of building your schema (code-first, SDL-first, etc), and it allows you to do full-schema transformations if you wish. An example of full schema transformation is the PgOmitArchived plugin https://github.com/graphile-contrib/pg-omit-archived which adds "soft delete" support to everything in your API with a minimum of fuss - by default soft deleted things are hidden, but you may specify for soft-deleted items to be included, or shown exclusively, using a GraphQL field argument.

> is it really more productive

I suggest you ask our users why they love us; I suspect you'll find the massive increase in productivity is one of the main reasons.

> is it robust, now that your business logic layer is intertwined with your API layer

As mentioned above, this needn't be the case; yes, it's robust.

PostGraphile maintainer here; what you’ve said is not really true for PostGraphile - we give you a huge toolbox of ways to customize, shape and extend your schema so it’s designed for consumers to consume rather than just being a way of exposing your database over GraphQL (which I agree is not necessarily the right goal in many cases). It does encourage you to put your business logic in the database, but you can also do logic in JS/TS if you see fit with our powerful plugin system and range of plugin factories for common tasks, such as makeExtendSchemaPlugin for adding your own types and resolvers. It constructs a GraphQL schema using the reference implementation so you can then use the schema with other tools if you like. For an example GraphQL schema that’s client focussed, see: https://graphile-starter.herokuapp.com/graphiql . I care deeply about building high quality GraphQL schemas; this is one of the reasons I contribute to the GraphQL Spec itself.

This absolutely isn't required; I think you might be getting confused with RBAC rather than RLS. Handy infosheet: https://learn.graphile.org/docs/PostgreSQL_Row_Level_Securit...

“While we will discuss how you can use the schema we create with PostGraphile, this article should be useful for anyone designing a Postgres schema.”

Agreed: an API that _just_ maps to CRUD operations isn’t good. I’m not advocating for that, neither is singingwolfboy, and the starter repo he’s linked to basically does not use them: there are only 4 CRUD mutations, all the others are custom. I rarely use CRUD operations in PostGraphile, mostly I use custom mutations either defined in SQL or TypeScript.

Absolutely not, this is a common misconception. Have a read of this: https://learn.graphile.org/docs/PostgreSQL_Row_Level_Securit...

It's not generally safe to expose SQL to untrusted clients. For example, PostgreSQL 12.2 was released yesterday and fixed a security issue where `ALTER ... DEPENDS ON EXTENSION` did not have any privilege check whatsoever. SQL is also not at all well suited for the needs of frontend web app developers - just ask Facebook about their experiences with FQL! Using an API that's more ergonomic for the frontend, such as GraphQL, backed by a language which is optimised for the backend, such as SQL, is the best of both worlds.

Check out Graphile Starter; it’s extremely batteries included! https://github.com/graphile/starter

Check out Graphile Starter; it is a fullstack starter project for building a SaaS with PostGraphile in TypeScript with React (Next.js) and Apollo Client. It has all of the modern tooling preconfigured including type checking, linting, formatting, unit testing, acceptance testing with Cypress, CI via GitHub actions, code generation to increase development velocity and much more. On top of that, the user account system is pre-built with everything you’d expect: login with username/email and password or social login with OAuth, forgot password flow, change password, multiple email management, email verification, etc. It also has a job queue and migration framework built in and preconfigured, plus instructions on how to deploy... I’ve put a lot of work into it! https://github.com/graphile/starter