The PostgreSQL documentation and the limitations of community
rhaas.blogspot.com
rhaas.blogspot.com
The reason is, I think, that flexibility is a pretty fundamental part of the project. We're great at providing building blocks (and documenting them), but we steer clear of describing a particular way to assemble them together.
For example, we might describe a particular HA approach, but then that would be perceived as "recommended / official" way, giving it preference over other (and equally valid) approaches and tooling. These "how to" docs are bound to be way more opinionated, so we just focus on documenting the pieces.
In other words, our docs are written by devs for devs, and we leave the higher level stuff to tutorials written by others etc.
The problems described come from trying to be everything in one place, but it does not have to be. As I understand you try to be mostly a data sheet, which is probably a net good, because it is the document needed to be maintained, even if hard to navigate.
However, there are more document classes that can be produced. Yes, a reference design is inevitably going to be opinionated, whether it is produced by project team or some internet person. A reference design produced by project team at least has a fighting chance at staying somewhat up to date. And one can discuss tradeoffs between different approaches in an application note.
A problem I have is so many tutorials, or 'best practices' I find on the internet are for older versions that don't really apply as well in newer versions of postgres. Like searching for logical replication, you find lots of information for pg_logical for older versions of postgres, but many of those parts are now baked into postgres, but with a different syntax, etc.
I would love to see a 'tutorials/guide' and 'best practices' part of the documentation that is updated with each new release, that give examples of the most common tasks, and when/why to use them, and when to move to something more advanced.
Some really basic stuff like "this is the 3 best ways to handle replication in version 15, and the 2 or 3 most common ways to do backups, or these are the recommended ways to migrate from the previous version either in place, or to a new server, etc.
Tutorials and best practices come about from real-world usage, which is why they're often out of date — by the time someone becomes enough of an expert on features specific to a version to write good-quality content of that nature, a new version is out. The only way to update those is through real-world usage, which means shipping pre-release versions for long-enough periods of time and having them used to ship real things, all before the release.
Does PostgreSQL do this? It hasn't in my experience, but it's admittedly limited to the last 5-6 years.
I tend to see this as duality of static/ephemeral. On one hand you have "vendors" treating software as rolling moving target - latest and greatest is all there is. On the other hand, there are community guides and tutorials that are out there and never retracted or amended.
Let's say I find an article that luckily states being written against older version of software. How do I know whether the information is still valid and applicable to a newer version? Well, if I am lucky and the vendor publishes growing changelog I have to read it and judge of the changes are applicable. Maybe I have to hunt down changelogs from all the interim versions just to see if anything important to me changed. Most probably I will find a very succinct summary of the change like "toll x was merged into tool y" or "updated interface of x". Now I have to try and find the discussion around the change in mailing lists (hopefully the team has not migrated to Discord).
Contrast this with hardware. Quite possibly I will find erratas and/or product change notices discussing the changes, maybe even application guides discussing how to apply changes. Unless there is an entirely new product line being released there is high change there will be rather detailed documentation on changes. Software is usually majorly lacking in this regard.
And no, versioned documentation is not the answer. Documentation gets updated not only technically, but also as discussed structurally. Once a piece of documentation gets moved to another section or even reworded I can no longer reasonably search for changes. Documentation of a changing, evolving thing is hard and interestingly this is where open source gets hit the most - there are almost no incentives for someone to write good documentation.
There are cases where PG is missing high-level docs where I don't think this applies.
For example, there's no official doc on how to write PL/pgSQL code. There's just an extremely-low-level language reference, covering each syntax element separately. There's no cookbook (other than the few examples per syntax element that exist to document the edge-cases of use of that syntax element); no tutorial; no efficiency/performance/scalability guide discussing when certain language features should be favored over others given the current way they're executed (e.g. is IF-ELSE, CASE-WHEN, or a series of IFs with early returns cheaper? when should I favor using FOR with a query, vs. when should I query data into an in-memory array variable and then use FOREACH, vs. when should I query data into a TEMPORARY table and then query that?); no place where you can get a sense for how procedure CALLs interact with MVCC (e.g. when they acquire + release locks, and therefore how and when they cause blocking on contended tables vs. how and when a SELECTed function that uses dblink/fdw to run independent txs would do so); etc. There isn't even a single mention of which PL/pgSQL exceptions are potentially raised by what PG builtin functions when called in a PL/pgSQL context; how to name those exceptions to match on them to catch them; or how to raise them yourself. I often need to dig into the PG source code to figure that out! (PL/pgSQL honestly feels, in docs terms, like a proprietary third-party language-engine "plugin" that someone bolted on, where the docs were expected to be provided by the third party, but never were. But it's not! It's a first-party language, and the reference implementation of how to create a language extension!)
It also forced the designers to think in those terms and to document the product from an overall perspective rather than a component perspective. It was high level enough to be useful, but not so high level as to be abstracted into hand holding tutorial exercises.
I feel like most modern software documentation entirely misses this component and would benefit greatly from having it.
"The Diátaxis framework aims to solve the problem of structure in technical documentation. It adopts a systematic approach to understanding the needs of documentation users in their cycle of interaction with a product.
Diátaxis identifies four modes of documentation - tutorials, how-to guides, technical reference and explanation. It derives its structure from the relationship between them.(...)"
Also.. early computer manufacturers like MITS had manuals in a similar vein for their Altair 8800 box but you can find many examples in this space, there's a stub of a Wikipedia page just for it:
I honestly prefer this type of documentation. ASP.NET Core has the complete opposite problem where it's too example based.
Combine that with maybe the info you are looking for actually is in the docs, but it's sprinkled across loads of examples so it's hard to find or build a comprehensive understanding of.
> But, you know, I was a loyal servant of the community process. I was asked to document that stuff, and I did, and I put it in the documentation in the place where it most logically seemed to go. The fact that the overall structure of the documentation probably isn't for the best is not my fault, nor is correcting it my responsibility. And it's not anyone else's responsibility, either.
...
> It's not difficult to understand why this happens. If I add a new feature to do a certain thing to PostgreSQL, I am the expert on that feature. There's nobody else who knows better than I do what the documentation for that feature ought to say. My work might have shortcomings just like anyone else's, but especially if I'm just adding new entries to tables that already contain dozens or hundreds of existing entries, how much difference of opinion can there reasonably be? It's more likely that reading the documentation will cause someone to take issue with the design of the feature itself than it is that they won't like the way it's documented.
Which, I think, is blind to the bigger issue.
As you note:
> The reason is, I think, that flexibility is a pretty fundamental part of the project. We're great at providing building blocks (and documenting them), but we steer clear of describing a particular way to assemble them together.
...
> In other words, our docs are written by devs for devs, and we leave the higher level stuff to tutorials written by others etc.
The deeper and more common problem between those two symptoms, which the OP misses, is that the people writing the docs often don't use, and maybe haven't ever used, the tool or features they're documenting in their most common productive modes. The most productive devs are often the least knowledgable people in how most, or even many, users use it.
Companies often hire (and compensate) someone to try to take the giant mess of dev-written reference content and make guides out of them. But if those people don't use the product either, you just get better-organized docs that still miss the point.
Most tools need usage experts writing docs far more than they need feature or software experts. The time of open-source tools' usage experts — "written by others etc." — is often as or more valuable than the time of the open-source project's engineers'. Usage experts are likely being compensated to do almost anything but document the open-source tool, or might even be compensated to document the tools privately or internally for others in an organization to use them better than potential competitors — the opposite of community.
The kinds of tools where this trends toward open tutorial creation and documentation tend to have communities of users who aren't as focused on specific tools or narrow usage as systems tools like PostgreSQL — gamedev and media production tools come to mind.
"Make a game" or "make a movie" are no less varied than "make an app" or "make a service", and can still be prone to tooling disputes (ie. using Unity vs. Unreal vs. Godot, Premiere vs. Final Cut vs. Davinci), but seem to fall into the trap of hoarding tool knowledge less often. Maybe because there's more authorship and recognition to those types of work, I'm not sure.
There is a great article talking about the 4 different types of documentation named the “document system”.
> The strengths of this process are also its weaknesses. A developer is, by definition, someone who spends the majority of their time doing development, which is to say writing code. Updating the documentation becomes a task that must be completed so that the code one has written can get committed so that one can move on to the next project and write some more code.
I may be misinterpreting, but I get the sense that the author feels that there is some kind of more optimal way to split up docs duties. IMO there is not. At least, not for reference docs. As the author said, the people implementing the code are in the best position to keep the reference information up-to-date.
If I grokked the rest of the article correctly, the author is essentially saying that the engineers have trouble writing and maintaining the other main types of docs [1] --- guides, tutorials, and overviews (explanations). It also sounds like they are having a "too many cooks in the kitchen" problem with pushing through changes to the other docs. I have a simple answer to that: hire some strong technical writers and make it clear to everyone that the TWs are Responsible and Accountable [2] for those docs. Also, make it explicit that the engineers are a Consulted role when it comes to guides / tutorials / overviews. Writing these types of docs is hard, specialized work. As the author said, the engineers have lots of other priorities. Of course, it's a bit self-serving for a TW to say "the solution is to hire TWs" but I get the sense that people don't realize that the easiest way to get good guides / tutorials / overviews is to hire people who have thought long and hard about those specialized tasks. If you want a good database, you don't expect your TWs to do the job. You get a database engineer. If you want good tutorials / guides / overviews, you likewise shouldn't expect your database engineer to do the job. You get TWs.
[2] https://en.m.wikipedia.org/wiki/Responsibility_assignment_ma...
And don't forget that there are valuable volunteers who are likely to go elsewhere if too many new rules are added that they don't want to live with.
Presumably, PostgreSQL has leaders who are responsible for steering the ship. If the project is going to succeed long-term, those leaders have to find ways to keep their contributors happy while also creating an organizational structure that leads to good docs. Easier said than done, I know, but it really is as simple as that.
Sorry if any of my comments came off naive or obtuse when it comes to open source dynamics. But the reality is that you need good docs, and I'm just trying to give an honest assessment from my experience of the conditions that lead to good docs.
If you want that apology to be meaningful, you should learn something.
When you're talking about a highly successful open source project that has been going for more than 3 decades, it is beyond ludicrous for you to say, "If the project is going to succeed long-term..." It already has succeeded long-term. And you would be better off figuring out why it works rather than lecturing about how it must work.
When you talk about "a potential model to draw from" for funding, please note that I've been involved with open source for about a quarter of a century. I've seen a LOT of funding models attempted. Mostly they run into one big problem. And that problem is that adding funding creates bruised egos because people say, "Why is he getting paid when I'm not?"
The one funding model that DOESN'T have this problem is when a company decides to pay its employees to work on features that it wants in the project. Now there are no bruised egos - the money comes from the company and it is clear why one person gets paid while another does not. There are still challenges with this model - employees are under pressure to get their contributions accepted whether or not the project likes them - but we've learned how to navigate those.
But now we're left back where we started. Companies who hire core developers don't generally need comprehensive documentation - they build internal documentation straight for their use case. So comprehensive external documentation is hard to find. Sometimes you'll wind up with things like an excellent introductory tutorial like https://docs.python.org/3/tutorial/. Usually, you don't. And generally it is hard to simply pay someone to take care of it for you.
Yes, your reaction here totally makes sense. Feedback acknowledged.
> If you want that apology to be meaningful, you should learn something.
I have re-read my earlier comments and I feel that you are being more hostile to me than is justified. I do not think you are adhering to HN's code of conduct guidelines for comments: https://news.ycombinator.com/newsguidelines.html#comments
> you would be better off figuring out why it works rather than lecturing about how it must work
This doesn't seem fair. The original post is about the limitations of the PostgreSQL docs. Docs have been the focus of my career for 10 years. I have experienced and analyzed docs problems in many contexts: small orgs, large orgs, open source, closed source. I made an on-topic comment about ways to resolve the problems that the PostgreSQL docs are facing. Is it the only solution? Of course not. But I totally have relevant experience in this domain and, just like you have a good idea about what generally works and doesn't work regarding open source funding, I have a pretty good idea about what generally works for creating the conditions that lead to good docs.
> So comprehensive external documentation is hard to find.
Again, I think the web platform space is relevant here. Web platform documentation could easily devolve into a tragedy of the commons situation. Yet MDN does exist and is an amazing resource.
Paragraphs 4 to 6 of your last comment seem to be arguing that hiring TWs is not an option for PostgreSQL. That is totally understandable. On another day maybe we would have arrived at that understanding on friendly terms and would have had a constructive conversation about how to create good docs when hiring TWs is not possible. But it's clear that my ideas aren't welcome here so I'll just stop now.
And, that person is being overtly hostile - I can only describe their behavior as making shit up in order to justify picking a fight and talking down to you.
I don't think you should take anything they say to heart - flag and move on at this point.
https://www.postgresql.org/about/sponsors/
Also, I think it's important to note that a lot of contributors aren't volunteering their "free time", they're being paid by some other employer to contribute to PostgreSQL as part of their job:
https://www.enterprisedb.com/blog/importance-of-giving-back-...
It isn't for contributing into a pot of money allowing some central PostgreSQL committee to hand out money for other things, like hiring people to do documentation.
From rough memory of an informal discussion with one of the members involved in the PG sponsorship area a while back, they had something like several hundred thousand US$ sitting around mostly unused.
The real problem is proposing something like this, and getting it okay-d. Seems like a tough (political) challenge.
There is a PostgreSQL core team (https://www.postgresql.org/developer/core/) that is a kind of steering committee, but does not make decisions concerning development. It consists of people from different companies.
The people who develop PostgreSQL are mostly hired by companies that have a commercial interest in the database and usually sell consulting, support, training, products that need PostgreSQL and customized closed-source forks of the database software.
There is no single authority in the PostgreSQL project. Decisions are made in mailing list discussions, cumbersome as that may be. The ultimate authority over the code is wielded by the committers (https://wiki.postgresql.org/wiki/Committers), which are highly trusted core developers who work for different companies. Development is slower than it might be with a single central authority, but that also part of the reason for the high software quality that PostgreSQL is known for.
Quote from Haas:
> But if on the other hand I propose some change to documentation that has existed for a long time, or some kind of structural change, there's a lot more room for disagreement. Because the change isn't strictly mechanical, the right answer is a lot more subjective. And because it's a change to existing content rather than the addition of new content, many more people will be familiar with it and have opinions on how it ought to be changed, if at all. Consequently, even when some developer does take time away from writing code to try to make some larger change to the documentation, it's often an uphill battle to get anything done, and people typically have to be content with small improvements.
Honest question: do PostgreSQL engineering decisions have the same dynamics as what's described here for docs decisions? If not, what is different about engineering decisions versus docs decisions? Is it just that engineering decisions can be literally benchmarked whereas docs decisions do not seem benchmark-able? Are there any other potential explanations for different dynamics between eng and docs?
If it does indeed just boil down to "docs are not benchmarkable" then I would suggest otherwise. You can create docs benchmarks. They won't have the rigor of engineering benchmarks but they at least establish some notion of docs quality and facilitate more targeted discussions during docs reviews. E.g. when there's a disagreement between an author and reviewer the author can say, "what docs benchmarks am I not following here?" The power of that kind of interaction is that you often do realize that the docs benchmarks are incomplete and some new dimension needs to be added to them. Or do the PostgreSQL contributor docs already have some guidelines along the lines of a "content quality checklist" and it's still not working?
A rigorous effort to survey the PostgreSQL community and get a deep sense of what the overall community considers "high-quality docs" can itself be a super insightful experience. Different developer communities often need / want a different focus in the docs. That can be the foundation for a fairly authoritative docs quality checklist.
Another thing that I'm very interested in, and will need to think through deeply some other day, is this notion that once you publish a doc, you can't touch it. It happens all the time and it's really weird.
While I think updating reference documents lends itself to subject matter experts (especially in a project where this approach is already successful), I think structuring them can be separated and given to a technical writer with more technical expertise or a developer with more documentation expertise.
We in infra-ops can give you more details about how our database clusters are designed for resilience, security, safety than you want on more levels than most people in the company know exist. We also have reasoning for all of this available. This is really good to have for customer question sets during sales.
However, this doesn't tell a developer how to connect his spring boot thingy to it, and how to connect and manage his service well. In fact, 80 - 90% or more of the things we know about our database are not relevant to a simple small-scale application running queries on it. And quite a lot of the issues you can have with running your application on a rock-solid database are entirely not relevant at a DBA level. Like, the database doesn't care if your DDL modification is backwards compatible.
And that's something we're currently learning together with a foundation team at work. They are documenting on how to use it well from their side, we're learning about easy mistakes to make and document those, and help with the actionable documentation. And in hard cases we kinda have to talk what's the plan, because it's usually not smart from a DBAs perspective.
And we have different angles of approach here. For example, we from infra-ops have setup a bunch of reusable and versioned template modules. These modules encapsulate all of the gritty consul/nomad/certificate handling and someone actually using our databases mostly needs to supply the module with the necessary environment variables their application needs. This in turn has been extended by a bunch of these teams to provide preset templates for spring boot, .Net, python, libpq so most applications just need to configure "I'm springdev and need to go to this database".
And currently another team is looking at building our own spring boot seeds, and/or using tools like cookiecutter or backstage to provide skeleton templates including most necessary boilerplate. We've started to maintain a template nomad job for simple applications as well.
With these things maturing, getting a simple service stood up can be done fairly rapidly if you know what you're doing. Setup the skeleton, throw out a couple of things you don't need, be done in an hour or so.
This however doesn't solve the initial problem. It's still tricky to know how to approach this stack and it takes time to either teach people how to approach this stack, or to condense all of this into simple how-to documents and/or workflows to get systems going.
https://documentation.divio.com/
We've been referring to it as the Divio system though, which appears to be the same thing.
LLMs are fit for this use. I think they could get trained on assembly language.
further, I think that at this point, the whole x86 spec is as impossible to analyze 'rationally' as any natural language. funny that it was created by dozens upon dozens of engineers in a rigorous and formal way; and the result is now mega complex. but I digress...
my point is to reconsider x86 spec as a 'natural' language which is the common "tongue" between software and PC hardware.
but by this point there's no expectation that a modern intel cpu is implementing the instructions in the way they were first created. it may well be emulating the x86 spec to the best of its capacity (another computer program "written" in 'microcode', a x86 virtual machine); so when thinking about x86 assembly spec as a language, the software computer is talking to the hardware computer; and if they agree the computer works. (so the socket and motherboard's buses can be understood as network devices xP )
finally, if I can come up with this (LLMs trained on assembly) so can any security researcher from any advanced academy or corporation.... meaning security researches NOT using LLMs like this ones I imagined are not doing their jobs correctly.
While there should be a lot of hesitancy at the implied assumption that any particular arrangement is at the efficient frontier -- most situations can be improved in most or all dimensions -- an exchanged loss in the kind of documents that pgsql-hackers is suited to producing is hard to replace.
And in the past they have come around on things like replication in core. So perhaps the future will bring some more handholding docs to supplement the reference docs.
[1] https://technicalwriting.tools/posts/playing-nicely-with-gen... (my blog)
https://www.postgresqltutorial.com/about-us/
is a very helpful supplement to the official docs.
Unfortunately it's not comprehensive - for example it doesn't appear to cover pgcrypto extension.
Here's a random snapshot of it from the Wayback machine:
http://web.archive.org/web/20040630081140/http://techdocs.po...
Much more user-level oriented than the reference stuff.
https://www.postgresql.org/docs/current/pgcrypto.html
The page contains the documentation of many functions, all of which raise the following error by default when you run them: No function matches the given name and argument types.
It turns out you first have to "install" them by running "create extension pgcrypto", which is obvious to someone who already knows postgres modules well, but not to anyone else, and isn't mentioned anywhere on the page!
Its like complaining your car manual doesn't say you have to open the hood of the car on the same page it explains how to change the wiper fluid.
I want to compute a sha1 hash in postgres. I make a google search, I click the first link. It says I should run `digest('mystring', 'sha1')`. It doesn't work.
Whereas it could be the following just by adding one line to the documentation:
I click the first link. It says I should run `CREATE EXTENSION pgcrypto` first. Then `digest('mystring', 'sha1')`. It works.
Here's an artificial example:
"Added a setting which allows you to change the size of a boba."
It doesn't answer any really useful questions, such as: why would you want to change the size of a boba? What is the effect of various sizes of boba? How does that interact with other settings?
As a database user, I actually want to know these things. Internally I have a mental model of how all these settings interact with the product, and I use information about the new setting to adjust that model.
For psql in particular, the documentation (as people have pointed out) shys away from anything too "opinionated."
But what it should do instead of have multiple opinionated examples. Multiple examples allow me to learn about the different tradeoffs and configuration options.
It reminds me of the old days, when the psql docs talked about optimization as a "black art", and basically said "it would be impossible to cover everything, so we won't cover anything."
Back in the Stone Age, companies like IBM had vast writing staffs. As a result, you got entire walls full of documentation. For OS/360, you got Concepts And Facilities, that explained what the software did for you; you got reference manuals; and Program Logic Manuals, which explained, in exhaustive/ing detail, how the program worked (with flowcharts!).
Arguably, the software I use today is far more complex than OS/360; yet we don't see the same attention to organization and detail in the documentation. I understand the reasons: IBM's dead-tree wall relied on paying tech writers, and that's incompatible with the tiny budgets most FOSS projects suffer. Far too often, I will go to a FOSS site's documentation page and discover a mass of links to pages that explain how to build the program on Itanium, or why this program is better than another one, with no walkthrough of “how to use this program for a typical case”, or even “if you want to do this kind of operation, these are the pages you need to read”. Often, the documentation isn't included in the repo, which lowers my confidence that the documentation and software are updated in step.
So I celebrate projects such as PostgreSQL, Emacs, and Arch Linux (for the wiki)—there are many more—where there is a real effort to create good documentation, even when I think that documentation can be improved or reorganized. Let's not allow the perfect to be the enemy of the good.
Fabrizio Benedetti did a cool analysis of various common docs-as-code architectures: https://passo.uno/docs-as-code-topologies/
It can make the difference in how long it takes you to complete a project at work.