Architecture.md | Hacker News Reader

Architecture.md

matklad.github.io

1,349 pointstodsacerdoti153 comments

Ah, we actually have one of these at Caddy: https://caddyserver.com/docs/architecture

(The filename on disk is literally "architecture.md" -- it is a Markdown file rendered by Caddy's template handler: https://github.com/caddyserver/website/blob/master/src/docs/...)

It could use some improvement, but it's been really great for helping people learn how Caddy 2 works at a high level. Beyond our docs, I always encourage new contributors to thoroughly explore the godoc and code: it's very well-commented and organized, especially once you know how it all comes together. A single document will never be sufficient. But it can help you map between concepts and code.

Code search is also invaluable for this; I recommend Sourcegraph: https://sourcegraph.com/github.com/caddyserver/caddy

Edit: One other valuable piece is explaining why the architecture is the way it is. Our architecture.md doc links to a video that explains how I arrived at Caddy 2's architecture (and why it's not arbitrary): https://www.youtube.com/watch?v=EhJO8giOqQs

Thanks for putting that out there. It's super interesting to see what we consider architecture to be. Your approach appears operational, focused on how the completed system functions, i.e. getting a new team member up to speed on the codebase.

Typically I try to start with tiers (1, 2, n-tiers...?) that show how the system might be deployed. I then list list layers (user/facade/business/data), interfaces between these layers, and components within each layer. I do that for each tier.

After that maybe something about quality objectives and how they might be met, eg: availability (MTTF / (MTTF + MTTR) * 100), efficiency, flexibility, integrity, interop, usability and so on.

This leads to a physical delpoyment model, which shows layers deployed to tiers. And yes, I'm rather fond of Visio.

Then a bit about approach (dev/deployment and operational management) risk and stakeholder management, technical reviews (change control board maybe?), and project reviews.

To be fair I come from a predominantly critical systems world (telecoms OSS and BSS, healthcare, transport and some fintech. And in that world architecture is very far removed from actual code until eventually.

How do you distill down critical outcomes of the architecture for people considering using your project? Based on my experience so far, engineers will look at a giant document, see phrases like “stakeholder management” and nope the fuck out.

What I want to know is, what are the key performance considerations, failure modes, recovery procedures, etc.

Critical outcomes are defined by quality objectives (I mentioned some above, others are reliability, robustness and portability).

People don't consider using my project. There's a client with a business problem, there's a vendor who solves problems for clients. The vendor produces an architecture document that describes how technology will achieve a solution †.

There is no noping the fuck out, as this is a hospital asking you for a one-off to manage/settle insurance payments. Or an electoral district asking you to merge three emergency response systems into one. Or Nokia asking you to tariff calls going through a switch in real time.

† This is nuanced. Often a client's procurement department invites a number of candidate vendors to submit proposals including a design proposal/architecture and associated cost estimates. Vendors range from the high end (McKinsey, Bain & Co, Ernst & Young) to the mid-tier (Wipro, Accenture and so on) to the niche.

> Or an electoral district asking you to merge three emergency response systems into one. Or Nokia asking you to tariff calls going through a switch in real time.

I’m well aware of what it is. I’ve been on real time telecom stuff (your last example) and know for a fact that engineers nope the fuck out of these huge ass architecture documents that include stakeholders, change control, etc.

Inevitably there is some kind of outage or botched upgrade with lots of finger pointing and then the vendor covers their ass by referencing “page 248 under heading ‘assumptions about bisectional bandwidth’” or some bullshit right before the section on ‘renegotiating requirements during a government declared emergency’.

There needs to be a better way because I assure you, the people using your system are (on average) barely going to skim your document.

Being a “very serious” industry does not change this. Look at the disaster that was Healthcare.gov. That had mountains of documents like you describe they overlooked the simple requirement of scaling identity lookups.

The people using my system never skim my document. Lawyers tend to scrutinize it. As for the document being a war-and-peace epic yes, this happens. Mostly because people cannot write technical documentation. The hugest system can be distilled down to a single diagram which should require no words of explanation.

By way of example - here's a conceptual design for an insurer wishing to participate in a panel of protection providers.

https://www.wittenburg.co.uk/images/consulting/conceptual_de...

If you're in this world all of it is self-evident, none of it a surprise, and you understand the flow. That document leads to other more detailed designs that each address one distinct rectangle in that image.

At some point all of this will become a specification. Again the immensity of that depends on the ability of the authors. On their ability to manage complexity, and their skill as technical writers.

Either way yes, when the wheels fall off that document becomes the truth, the one everyone goes to to settle disputes. And with a capable team it never gets to that because the documentation is succint and agreed to, and what is delivered matches what was specified.

The way to solve that is basically DevOps. DevOps being a solution to a business problem, it is implemented as an array of "features" of integrating teams that work together on a product. It's a multi headed hydra and it's not easy. But the end result is people communicating better, which is the only thing that can really address a developer noping the fuck out. There is no technical thing, no document, no policy, etc that can fix it, as far as I'm aware.

"... the disaster that was Healthcare.gov"

OMG you worked on that? The blame for this one (and Universal Credit) sits with GDS. Their approach at the time was agile or die, and then try to develop an enterprise system like they did the Guardian's CMS (GDS was staffed initally by ex-Guardian devs).

I had the misfortune of reviewing their Government Gateway replacement, which at the time focused singularly on identity federation. GDS was hugely anti-Microsoft, who built the Govrernment Gateway. And so they chose the SAML protocol over the incumbent ADFS, breaking every dependency.

GDS was way out of its league. An inexperienced 25y/o will draft exactly the kind of architecture document you describe.

I feel for you, and I now understand the cyncism.

Your approach seems like it would reduce the likelihood of implementation-time surprises. It also seems like different sections would satisfy different audiences. Do you have any book recommendations?

Two books I that were instrumental for me to learn how to explain the architecture of my software clearly to other people were "The Architecture of Open Source Applications" Volumes I and II (http://aosabook.org/en/index.html)

I do indeed. Here's a complete list of books I recommend when asked -

https://www.wittenburg.co.uk/Work/Books.aspx

The one that made the biggest impact on my approach to architecture is "Designing Solutions for Your Business Problems: A Structured Process for Managers and Consultants" by Betty Vandenbosch.

The one that made the biggest impact on me in ever is "Wicked Problems, Righteous Solutions" by Peter DeGrace and Leslie Hulet Stahl. Lots of forehead slapping and kicking myself ensued. Even more applicable today than it was in 1990 when written.

"Peopleware: Productive Projects and Teams" by Tom DeMarco and Timothy Lister comes as a very close second.

Not directly architecture or technology-related is "The Story Factor by Anette Simmons". It... changes you. Really useful for technical writing.

Thank you.

Thank you for the book recommends

Also i read your home > work articles

They are excellent

Off topic - what is the theme used in Caddy doc site? looks super cool.

Matt's Custom Theme? :) I dunno, I haven't named it. Glad you like it!

I have a similar advice, but I will go one step further: add README.md to other folders as well. It is dope to have a map of your whole system in an Architecture.md (or a README if it's not too long), but it's even more dope to be able to click through it and have submaps of how other components are structured.

Displaying the folder/file structure and explaining what is what is a must. An example from Diem[1]:

  consensus
  ├── src
  │   ├── block_storage          # In-memory storage of blocks and related data structures
  │   ├── consensusdb            # Database interaction to persist consensus data for safety and liveness
  │   ├── liveness               # RoundState, proposer, and other liveness related code
  │   └── test_utils             # Mock implementations that are used for testing only
  └── consensus-types            # Consensus data types (i.e. quorum certificates)
  └── safety-rules               # Safety (voting) rules

I recently digged into dependabot's code, and I found it extremely well structured. For example you have an Architecture section in the first README[2] with a diagram (how awesome is that!) and with links to the README of all the sub components, which themselves live in the subfolders[3].

What I dread the most is going through a new codebase and not seeing any documentation in internal packages. Like how the fuck am I supposed to understand anything in there? By reading all the code?

[1]: https://github.com/diem/diem/tree/master/consensus#how-is-th...

[2]: https://github.com/dependabot/dependabot-core#architecture

[3]: https://github.com/dependabot/dependabot-core/blob/main/comm...

> Like how the fuck am I supposed to understand anything in there? By reading all the code?

This is one of the superpowers of Go: for most Go projects, this is exactly what I'd do. Just read the code. It's easy to follow, it's all formatted the same, very little implicit behavior, and I don't need an IDE to do it.

Few languages were designed to be read by others. Thankfully Go is one of them.

I am afraid, this doesn't quite cut the mustard. Code just can't replace a human readable Architecture diagram + explanations of whys of choices and hows of the system. Programming language doesn't replace this no matter how clear and modular it is. While Go is nice, you're vastly under-appreciating architectural documentation.

My standard argument regarding the criticality of documentation: no amount of code can ever explain what isn't there.

What optimizations have you tried that failed? What 3rd party tool used to be integrated but now isn’t because the maintenance burden was too high? Why is it safe to ignore this exception?

> very little implicit behavior

Actually I find it quite implicit the way Go structures packages. You definitely need an IDE (I use neovim + coc.nvim) to jump to a definition unless you want to grep the folder for where a struct is defined...

There's one issue I can think of, if the struct is defined in the same package then it's a bit of a pain to find which file actually defines it (in Rust you have to be explicit, each file is a module).

But that's it I believe, if it's in a different package then you'd have to explicitly write the import

I agree that Golang has the easiest code to read, and jumping into a Golang project (even massive) is always a happy experience.

But still, you can't replace documentation.

When I first started programming, I thought the description next to the folder / file name on github was actually describing the item - as you did above, and not just the message from the last commit that altered the file.

Many years later, I still believe that's how it should be.

Agree! I gave that feedback many times, who cares about commit message? I want documentation here!

I think Github should really start taking a stance on convention, and come up with project structure conventions that would help project navigation. For example, youtube does that with chapters in videos, if you write a description with timestamps it will display these chapters in the timeline of the video[1].

IMO Github should do the same if you have a README in your folder, describing the folder with a certain syntax, then it should extract this info and show it next to folders.

If anyone at Github is reading this, pretty please :D?

[1]: https://www.youtube.com/watch?v=XfP862hCrDM

I’ve recommitted a number of times just because the last commit message was dumb or useless or worse - wrong.

This makes me think we should be able to put something like a standard tag for a 1-line summary (maybe embedded in some markdown in a comment) in a fashion the VCS's can detect & extract and then render in their UXs.

e.g. something very simple would probably work like the first line found that matches something like:

   [Ignore_Non_Words]Summary[IgnorePunctuation&Spaces](Capture)

You could probably solve most use cases by just pulling the first line from the readme in that folder.

It would encourage people to write folder-level readmes if nothing else.

I think you could get a lot of mileage from using existing formats in code and comments.

So for instance, if save_parser.py starts with

    """ Parse content loaded from save files

    ...

    """

and math_backend.rs starts with

    //! List of data structures necessary for maths operations
    //!
    //! ...

then your source tree UX should look like:

    src
    ├── save_parser.py      # Parse content loaded from save files
    ├── math_backend.rs     # List of data structures necessary for maths operations

This will help people browsing code in github, but folks browsing code in their IDE will most likely be lost, as the documentation you advice to put into "README.md"s should, IMHO, be in the module/package documentation in the source.

One can still link to those from the global README.md

READMEs and API docs aren't mutually exclusive. But they cover different scopes. READMEs are more suited for high-level documentation, focusing on the /why/ instead of the /how/.

I'm not sure I understand why someone in their IDE would be lost, you can't read README.md and see the file structure in your IDE?

Documentation in the source is different from a directory organization documentation IMO. Rust has a lot of tools to write good documentation in source, and it is here to help produce good documentation for users of a library, not to explain how to contribute to the code or understand how everything is architected.

> I'm not sure I understand why someone in their IDE would be lost, you can't read README.md and see the file structure in your IDE?

Of course one can see the files. But I rarely deal with "files" in my IDE, I deal with packages/namespaces/modules/functions/classes/methods. The natural place to put architecture documentation would therefore be the top-level code unit (e.g. package/module/namespace), to which one can easily navigate when viewing docs.

> Documentation in the source is different from a directory organization documentation.

Is it, given that directories tend to be organized around code units (again: packages/namespaces/modules/classes)? It is thinkable to store code in databases (see Dylan's IDE, either implemented or it was on the roadmap, I don't remember) and all the architecture documentation not directly attached to semantic units of your code would be lost.

100% agree. A very good implementation of your ideas can be found in the Redis codebase. Most important files have a long doc comment in the beginning.

See https://github.com/redis/redis/blob/unstable/src/ziplist.c for an example.

your example is a comment of the code/implementation, related to the actual file, not to the architecture

I still don't think this is a good idea, after having worked a lot with Golang and Rust doc, the documentation generated from comments is targeted to users of the library, not contributors.

(And Rust/Golang have the best documentation system out there of any language I have ever seen).

For example, documentation in Golang does not follow your file structure, it just pulls up any public API comment and displays it. It makes sense, why would I care about how these functionalities are implemented and architected as a user?

Second point: a system has different types of languages and ways to organize things, so it’s not always clear where to look for doc. Whereas a README is always obvious.

Any well known tools which print above like folder path trees command-line?

On Linux there is a nice little utility called “tree” https://packages.debian.org/stable/tree Not sure about Mac or Windows.

tree

brew tree (if you are on Mac)

I also use such hierarchical style, with lower level REAODMEs refferring iddividual source files. So that user continues from READMEs to code comments.

Example: https://github.com/cl-test-grid/cl-test-grid#system-architec...

On every level I first explain the task solved by this module and then go into the implementation details. Knowing the purpose first helps reader to understand impl details.

As someone who is in week two of spooling up on a multi-million-line codebase where most of the original authors have moved on to other projects, please, I beg you to heed this advice. I spend the vast majority of my time figuring out where a change needs to happen. The patches themselves are no more than 10% of the work.

(The other significant factor is running tests.)

Well, at least it sounds like you have tests :)

Good point. :)

It's called job security bub. I'm not going to write out everything you need to know. Hire an expert and whatever time it takes him to figure out is how much it's going to cost you to churn through employees. Don't like it? Be a better employer so your employees don't leave.

Lovely - now when you get in a car accident, your teammates get to be upset by losing you and have to reverse engineer the mess you left. Hope you don't plan to ever take vacations either - you're too important to leave.

I’m not “bub” to you, or to anyone else. I’m also not an employer, but if I were, I would absolutely not hire anyone with your attitude.

I suppose you're being sarcastic but I certainly have met people with this attitude. Sometimes its not explicit or intentional either, just the lack of time/importance from the business side of things that creates a situation where things are poorly documented/understood and it creates a dependency on certain people/companies. Ultimately I feel its up to us developers to stress the importance of investing time on documentation, but also be mature enough ourselves to realise that this is part of the job.

I love architecture docs, but find they're often written using a funny process:

  1. Spend a long time writing the doc.
  2. Wait for a person to chance upon it.
  3. Hope you anticipated their questions.

It seems like the most important thing a person can do is reverse this:

  1. Say who the doc is for.
  2. Find that person. Ask them to try a lil contribution.
  3. Frantically write / revise the doc.

IMO it's a lot like creating a presentation. The earlier the feedback the better!

Yep, feedback is valuable. And the earlier the more valuable (it's like NPV...).

I love coding so much, and find it really hard to express ideas in natural language, so that in the end documentation... doesn't happen as much as it should.

What I find that really helps is the following:

1. Write down architecture specs (with interface specs etc), before coding. Not bloated, but really minimalistic. 2. Review these ideas with peers. 3. Happy coding and refine the docs.

I like this idea a lot, but you will cause a lot of people to bounce at step 2. Or at least, that has been my experience over the years. No matter how much you reassure them that it is okay if stuff is confusing and in fact you'd like to know about it so you can fix it, they'll say "great" and then go radio silent 99% of the time.

Sounds right to me too. One quicker way to improve the 'first process' is to change only step one - do not spend a long time, but instead write a few paragraphs with what's most important and/or top-of-mind. Often, this opens the door to more contributions and questions.

Of course, update accordingly whenever you find yourself in a discussion about something with a contributor (no matter if the architecture doc is even part of the discussion or not).

I feel like for any long-running project, that person is at least ME. If I haven't written down some architectural information for complex projects, when I revisit a project after it being dormant for half year, I need to poke around to figure things out again.

If I have written down architecture notes in the first place, they are very helpful at this point; and if I haven't, it's a good time to start because I'll be acutely aware of the non-obvious parts as I re-familiarize myself with the code.

I usually reach for a friend, or someone I've met before, since using the first version of a doc is asking a lot! (And they're often part of the target audience).

A question can be seen as a bug reported against the documentation...

Apart from answering them, I end up converting 15% - 20% of questions into some revised content of my docs.

Most documentation follows that first path. With a step 4. that is basically "only update this when something is broken, or when we're hiring someone new".

Doing documentation well, especially if you're working on brand new tech, is very frustrating and difficult. You're often moving too fast to find the time to retroactively update documentation, and you're right back into the viscous cycle of it constantly being out of date. I don't know what the solution is.

This is also the idea behind Readme Driven Development. See https://news.ycombinator.com/item?id=25222601

I read the title at HN and I was like "okay seems interesting" then I saw "matklad" and went "Holy shit, must be great stuff".

I know I will sound another Rust evangelist, but people this person is the main maintainer of RA (Rust-analyzer), a LSP protocol implementation, anyone who tried RLS (Rust Language Server) then RA knows how great this tool helps you at learning and developing stuff with Rust. I use the nightly version (which updates everyday) and ohh boy... Never had the "opportunity" to caught a nasty bug or anything.