Polylith is a functional software architecture at the system scale
polylith.gitbook.io
polylith.gitbook.io
Anyway, congrats with the project; it seems pretty common sense idea wise so it is nice but cannot see many people jumping through the hoops and adopt it; people are too opinionated (including me).
I produce a bit of music and one thing I can never remember how to do without looking it up is side chain compression. I don't use it very often but when I do, I always have to Google what the controls / routing is to be able to do it.
In the beginning it was forum posts, maybe with a couple of images to help illustrate where to click etc. Now it's 10 minute YouTube videos replete with needless introductions about the person creating it and hat in hand begging for likes or subscriptions. You have to go digging to find what you're looking for.
I've been experimenting in this space a bit though (weirdly enough with video game tutorials). I'd be interested in how others find this format for learning or understanding concepts. Here's an example
https://apex-fundamentalists.vodon.gg/videos/0e074984-f9f0-4...
We decided to include both videos and text in the documentation because we know that people are different, and have different preferences for learning.
You're right that people are opinionated, and that it's difficult to convince them to try new ideas. However, I've been pleasantly surprised at how open the Clojure community has been to our concept and at the momentum it's starting to build.
It'll be interesting to see if other language communities have the same mindset.
I think for this reason the Lego analogy needs to be completely abandoned. Every effort of software modularization has been analogized to Lego at some point, and yet they all fall short. No arhicture claims they are the equivalent of twine glue and toothpicks, even if that's what it ends up being. So any N+1 language/framework/architecture promising it's going to be Lego (for reals this time), even if it does actually deliver, will not be taken seriously when they make the claim.
https://storage.googleapis.com/download/storage/v1/b/gitbook...
This is because the video says how nice would it be to have a banana without a slightest description of what the banana is and why do we need it. Absolutely useless video. And yes I very much prefer reading a document for understanding concepts.
As for the project itself. I did a very brief reading and I fail to grasp any particular value. Basically what I understood is that if you organize your code in some particular way we will do some particular things.
Maybe if they've presented clear example I would be willing to dig in deeper. For now I've "invested" couple of minutes and watched that "nothing" video and I do not think I'll invest any more time. Maybe it is my loss and their idea is amazing but they definitely do not make it easy to understand what's in it for me.
If you watch it at 1.5x speed, then it'll take less than 7 minutes of your time.
If you're still not getting the "why" of Polylith after this video, then I'd be very grateful if you could give us some quick feedback on what you're missing. That will help us figure out how to explain the concepts better in the future.
The video spends too long telling me things I already know - e.g. what are functions / objects / layers / etc.
Tell me something I don't know - such as what makes Polylith unique. The video says this is "Components". Well what's so different about those? They sound like microservices or perhaps high-level objects. Or perhaps interfaces.
Show an example!
Components do have attributes in common with microservices and with stateless objects (e.g. a public interface and encapsulated implementation).
Where components differ from microservices is that a component's interface is simply a collection of functions, rather than network-facing endpoints. This means that multiple components can be deployed into a single artefact, keeping deployment complexity and cost down.
Where components differ from objects is that a component is a higher-level abstraction, closer in scope to a microservice.
However, we think that Polylith's biggest differentiator is the separation it gives between development and production. Let's say you have a Polylith project with 100 components, that you deploy in production across 10 services. You can work with all 100 components in a single development environment, and test them as though they're a monolith, even though they're not deployed that way in production. It's a lot like building systems with LEGO, and we think its just as fun!
Is this a "collection of functions" or "collection of function declarations"? If you want to communicate concepts this is not the best example. And for either case many languages already have the solutions.
I agree with you that many languages have syntax to support building interfaces. However, I've yet to come across one that offers all the benefits of Polylith's approach with components.
My lines about the video part was exactly about this particular video.
Did my response to `cjg` in this thread help to clarify anything for you?
In my ecosystem I simply do not really have problems that you suggest we do and the functionality that you relegate to that "component" idea is not something new and unavailable.
I work mainly in C++, Delphi/Lazarus, JavaScript. Many others as well but not very often. C++ and Delphi handle the domain just fine. Javascript - bit less convenient still easy.
There are three approaches that I'm aware of for solving this problem: 1) copy/paste the shared code between the two repositories, 2) freeze the code into a library that both services depend on, 3) keep both services in one repository, extract the shared code into a module or component, both services depend on the shared module, deploy the services as separate artefacts.
1) is bad for obvious reasons, 2) adds unnecessary friction to the development process, and 3) is how Polylith solves it.
I was wondering if you've come across another way to achieve 3), or perhaps a fourth approach?
Pattern goes like this.
1) Client hires me to develop product. After initial phases I determine what parts (code) of other products I can salvage for reuse. At this point I either copy those parts or get the latest version if those parts are 3rd party libraries which at this point is basically the same. So in between different projects I always use approach (1). You are free to laugh at me. I develop for 40 years already and this approach saves me from countless headaches related to "purism". Unless there is really really big reason I do not want to change something in a piece of shared code and then test countless permutations in unrelated projects. Thanks but no thanks. Also once my product reaches maturity I usually transfer it to client.
2) Stage 2 - working on a single product. Even though I avoid "microservices" as deployable like a plague the product still might consist from few physical executables / services. In this case the "components" code is shared as a code and the code uses Interfaces (or simulation in case of Javascript for example) when I feel that I need to abstract some "component" so that I can replace it. so this is your (2) and it poses zero friction to me unlike what you claim.
Types of products I develop ranges from firmware, to game like native applications with device control and accelerated graphic and multimedia, to enterprise backends etc. I have way bigger things to deal with rather than nitpicking whether concept of Component through Interface / library / etc. poses mental / maintenance challenge for me (hint it is not).
I'm still intrigued to understand exactly how you share code across services. Let's say that you've written a piece of code for logging, which you want to use in both service A and service B. Do you package it up in a library? If so, doesn't that mean you have to place that library in a repository, so both services can access it? Doesn't that mean that if you want to make changes to the logging code that you now have to publish a new version of the library, and remember to update both services to depend on the new version?
That's the friction I'm talking about.
With Polylith, the logging code would live in a component that's directly accessible to all the other components in the system. That's because Polylith lets us work with all our components as if they're a monolith (even if we chose to deploy them as multiple services). This means that when we update the logging component, there's zero friction to update any impacted components in the services.
If the change only affects the logging component's implementation (and not its interface) then no other components need to be updated, and we can just redeploy the system. If it's a breaking change to the interface, then we can immediately fix the impacted components within our monolithic development environment. If the change is a refactor of the logging component's interface, then the other components will be automatically updated by our refactoring tool!
Hopefully that explains how Polylith solves this challenge so elegantly.
I package it mostly as a code. For example in C++ those would be the header file "xxx.h" ans an interface and "xxx.cpp" as implementation. If I only change the implementation there is no need for me to to touch anything else. Build system will figure out what services (executables) need to be rebuild and relinked. It will then build, deploy, run tests etc while I sit and pick my nose.
Is it because they’re “video native” and know how to navigate it better? Surely they’re not sitting through 2 minutes introductions. Do they inherently understand how to return to the important parts of a video for future reference? My brain just does not operate like theirs, it seems.
But following your idea of finding a reason that assumes it's actually more beneficial learning from video than text for people in that class, you need to consider two alternatives.
The first is that they are better at video, but the second is that they are worse as long form text.
Polylith has been gaining momentum within the Clojure community. As an example, you can follow along with Sean Corfield's journey migrating the World Singles codebase to Polylith: https://corfield.org/blog/2021/02/23/deps-edn-monorepo/
There's an active Slack channel, with people and teams using Polylith on both commercial and side-projects: https://clojurians.slack.com/archives/C013B7MQHJQ
Outside of the Clojure community, there's a project to port Polylith to Python, using Poetry: https://pypi.org/project/poetry-polylith-plugin/
We'd love to see it get ported to other languages too!
A couple points though:
- i wish you had made it clear from the start that the architecture applies to individual applications / services.
I wondered if "components" were made to represent only libraries of functions, or if they could be their own server running along side other services. You need to wait for the very end to get the idea that a polylith distributes / microservices architecture means "many polilths next to each other".
Which means that polylith might help organize the _code_ of a microservices monorepo, but will be of no help with the _execution_ problems of it (eg where should the data live, how many services do you have to traverse, how do you deal with desynchro and caching, etc... In my experience those are the thorny problems, not so much how your libs are laid out - but maybe that's because we already use similar ideas coming from "clean architecture", "hexagonal", etc..
- I am right to understand that most code in "components" should be concerned mostly with business domain, and free of "big" frameworks (spring / rails / etc..) and that only "bases" should be concerned with that ?
- what is the story when you need to fake a component that represents a distributed service ? The "component" folder will have an API,and an implementation that targets the real service ; but should it also provide a "fake" implementation ? Or would it be a different component sharing the same API ?
- Are there sides of the architecture that depends on being done with a dynamic language like closure ? Have you actually tried it with other languages ?
Thanks !
- The main idea is to allow bricks (as we call the "LEGO bricks"
- components and bases) to be put together into projects from where we generate artifacts (libraries, tools and different kind of services). Maybe we should have a sentence like that on top of the first page of the high level doc!
- The help the Polylith architecture provides when it comes to how to execute the code is that it makes it much easier to reorganise the code in production (by splitting, merging, or creating new services by reusing existing components) without affecting your single development experience (which is always only one single project). This is where the LEGO idea comes in. But you are right, it doesn't help you at all with the implementation of the code.
- A component can handle business logic, e.g. some part of our domain, others might manage integration with external systems, and others will be responsible for infrastructure features such as logging or persistence.
- The way you introduce a "fake" component (a component that can replace an existing component) is to create a new component that implement the same interface. So yes, it should be a new component that implements the same "API".
- We haven't tried it with other languages than Clojure, but the idea is applicable to other languages because it's not about the language syntax. David Vujic is working on tooling support for Python: https://davidvujic.blogspot.com/2022/02/a-fresh-take-on-mono...
I believe it would be beneficial to supply more example projects. I found one, https://github.com/furkan3ayraktar/clojure-polylith-realworl..., however, it uses SQLite. Maybe an example which uses Postgres, and Redis for caching would be more real-world? Also, maybe a few deployment examples? Heroku, AWS, GCP, Kubernetes?
One question I have, how are ENV variable driven configurations handled? For example, if I need a `DATABASE_URL` etc. I recently ran into an issue, https://discord.com/channels/313110246643990528/313110246643..., in my own Clojure web service attempt where I could not use `def` to define the individual variables since they are evaluated at Uberjar build time. I eventually converted it to a `defn` but then it gets evaluated every time it's used.
I invested about 40 minutes into the video in TFA and another 10 minutes in the "nutshell" video you linked. I haven't heard of Polylith before today. I really enjoyed your presentation and I thought it was of a very high quality as far as the amount of practice you put in and the visuals you created to accompany your words.
However, I have to be honest and say that the quality of the presentation is really the only thing that kept me watching. I figured if you spent so much time on the presentation there must be something here worth hearing. although I did want to abandon the effort and move on at several points because I just had no idea what you were trying to convey.
Or let me put it this way: I did know what you were talking about, but I continually felt like I was missing something because what you were telling me was obvious to me.
In the 40 minute video some notes I had in my head were that:
- First of all, you need to cut out the guy who immediately says he forgets your name. That's not a good way to start off the video.
- But more importantly, you only start talking about Polylith at minute 8 of the talk. Really, I had to double check at a couple points to make sure I was watching the right video because you spent most of the first part of the presentation talking about a different project called Code city, and I started to think this was a presentation on that. What you do at minute 8 you should have done immediately.
- When you do finally introduce Polylith, you don't tell me anything specific or interesting. The words you use are quite generic, and would be used by almost any project to describe itself. All projects want to be simple, flexible, and productive.
- At 12 minutes in things started to click for me when you actually started getting more specific. I don't know what your YouTube statistics look like but I'm willing to bet by this point you've lost almost all of your viewers.
- At 27 minutes in I got really excited because the MC came back to say that we were going to move past concepts and theory and get a demo of the actual product. But what follows is not a demo at all, but further descriptions of a system.
- Finally I'm struck that you introduce your project Scrintel, which is described as a video transcription service, but I'm not actually using Scrintel as I watch your video. Instead I'm using YouTube. This is the perfect opportunity for you to demo your actual software. I literally had to look at the video captions to understand what the MC was saying, and the captions I believe are wrong. Why aren't you using your transcription software to transcribe your video that is selling me on your transcription software?
Moving on to your 10 minute nutshell video, it's more of the same. After spending almost an hour exploring the content you've posted about your product, you've actually never demonstrated your product to me once!
One thing about the nutshell video, and your videos in general, is that I think you should abandon the Lego metaphor. First, it doesn't have the intended effect when you display the toothpick Whitehouse and the Lego Whitehouse. When I see these two, you want me to think about how difficult the toothpick version is to make, and how we shouldn't construct things that way in software, but all I can think about is how detailed it is compared to the Lego model. The toothpick model isn't a mess, it's a masterpiece. I admire the person who made it, but you're telling me that I shouldn't want to build software with that model.
Secondly, the Lego analogy is overdone. As I said in another post, every architecture has made this claim, even if they are actually toothpicks. So when you make it, it's going to be met with skepticism, and since you don't actually demo any of your software in these videos when you make the claim, you're going to be dismissed. Especially when your videos rest so heavily on the metaphor. My opinion is, if you take every instance of describing Polylith as Lego, and you replace it with a demo of your project showing how it's better in concrete terms on a real-world example (not your startup), you will be 1000% in a better place.
I'll leave Joakim to respond to your comments about his presentation, and I'll respond directly to your comments about the nutshell video.
> you've actually never demonstrated your product to me once
That's partially true, although I'd argue that by showing the components and bases from an example system, we are showing "the product". Perhaps you're thinking about showing code or filesystem structure from a complete production Polylith workspace? We do that in other videos, but didn't think it would be a good fit in the 10-minute "why?" introduction.
> The toothpick model isn't a mess, it's a masterpiece. I admire the person who made it, but you're telling me that I shouldn't want to build software with that model.
I agree that the toothpick model is a masterpiece! However, the reason we shouldn't want to build software in the same way comes down to one word: change. The real Whitehouse doesn't change shape very often, but all the software I've ever been involved in building does. That's why building software with LEGO pieces is so much better than building it with toothpicks and glue. Which is exactly why it's a such strong metaphor.
> Secondly, the Lego analogy is overdone. As I said in another post, every architecture has made this claim, even if they are actually toothpicks.
You might be right about this, but most of the other recent architectures we'd come across seemed to steer away from talking about LEGO (onion, hexagon, DDD, microservices, etc.), so we were hoping it wasn't overused.
- The Code City project is a cool way of visualising code and the idea was to show how we normally organise code today, and what problems it creates (development experience + sharing). I agree I could have been more clear about that.
- This is the longer video where I (and Furkan) try to explain what Polylith is, what problems it solves, how it works, and why you should use it. The sad fact is that it's super hard to explain Polylith (we have given about 15 different presentations) and I think you need to try it out yourself to get an idea how it is to work with and which problems it solves.
I will leave to Furkan to answer your questions about Scrintal.
For example, because of the forced conventions, it's trivial for the Polylith Tool to perform static code analysis on a Polylith codebase. This means that the tool can identify the subset of tests to run, based on the components that have changed since the last run. Which leads to a fast-feedback loop, and encourages both good testing practices and fine-grained component modularity.
Polylith gives you a system-level architectural building block; the component, which encourages a modular design and separation of concerns. However, you're right that it's still possible to create spaghetti code with Polylith. All it would take is poorly designed components, with bad names, multiple reasons to change, and exposing their state everywhere. However, I'd argue that when you give someone a well designed tool (like Polylith), then they're more likely to craft a well designed product with it.
To understand how builds and deployments work with Polylith, I'd recommend reading the "Workflow" section in the Polylith Tool's documentation: https://polylith.gitbook.io/poly/workflow/shell (especially "Build", "Git", "Continuous Integration", and "Testing").
I'm sort of reminded of adopting a formatting and linting stack across a codebase - so long as it mostly makes mostly good choices, the shared conventions are usually a net win overall just because it (a) makes code more accessible across the team (b) gives you a solid default way to resolve a lot of choices.
The whole "can -reliably- figure out which tests need to be re-run" part specifically sounds like it would be a very nice thing to have.
I suspect your biggest challenge in terms of adoption will be the requirement for development group wide buy in to get the full benefits, but that's a problem that's kind of inherent to the goals you're trying to achieve here and so I shall simply wish you good luck with that part.
Including my favourite; a complete untangling of your development and production environments. With Polylith you always develop your system as a monolith (because that's the most effective way to build software), but you're able to deploy it as multiple services (because that's sometimes the most effective way to run software). It turns out that separating deployment complexity from development complexity is a game-changer, and something that I haven't come across from other architectures.
It's true that you don't reap all the benefits of Polylith until your entire codebase uses the same structure, which feels a bit like "all or nothing". However, many of the benefits are unlocked "as you go", so even converting one or two existing microservices to Polylith will feel like a nicer codebase to work with.
> so I shall simply wish you good luck with that part
Thanks!
Having said that, I'm sure it's possible to run the prod version(s) locally and test them, but then don't you lose the benefits of separating deployment complexity from development complexity since you're now running the prod version locally anyways?
Hypothetically, say I'm big corp A with my thousands of developers and hundred of micro services that are just polylith projects. Further, let's say, I have N services that depend on component B. Now, if one team needs to make a breaking change on component B (say you need to change the interface), how would you suggest handling it based on polylith architecture? Would you version each component so that services can pin the component version? Or would you create a new component? Something else? Intuitively, versioning sounds like a mess of thousands of repos. On the other hand, creating a new component would create precedent that might be used to justify an explosion of components that may make your workspace a mess of almost identical components. While refactoring sounds like the way forward here, if you've dug yourself a hole with bad design choices then polylith seems like it would give you more rope to hang yourself with. Otherwise you have to coordinate with all the teams needed to figure out how to modify the N services depending on component B. With typical microservices, my understanding is that this wouldn't happen so long as the service's API remained constant.
You are right that you need to handle breaking changes in some way. Refactor all the code that uses the changed component is probably the best way to go in most cases because it keeps the code as simple as possible. Second best (or best in some situations) is to introduce a new function in the existing component (if it's not a huge change, then a new component can make sense). This can be done in different ways, e.g. by adding one more signature to the function or by putting the new version in a sub namespace, e.g. 'v2' in the interface namespace.
You will face similar coordination problems with microservices too. I worked in a project where we had around 100 microservices. To share code between services we created libraries that was shared across services. Sometimes we found bugs due to some services used an old version of a library. Then we went through all the services to make sure they all used the latest version of all libraries, and that could take two weeks for one person (full time)! You had to go and ask people about breaking changes that was made several weeks ago or try to figure it out yourself.
The alternative is to not share any code and just copy/paste everything (or implement the same shared functionality from scratch every time) but that is probably even worse, because you will not get rid of the coordination needs and if you find a bug in one service, you have to go through all code in all 100 services manually to see if any of the other 99 services contain the same bug, or hope for the best if you don't.
For example, "Polylith in a Nutshell" mentions (pure) functions, but they never come up again.
It's tricky to pick the right level of detail when explaining an idea like Polylith. If we could precisely identify the level of understanding of the reader, then we could tailor the content to perfectly match their experience level and vocabulary. Unfortunately, we can't do that, so we have to pick a middle ground, where we probably over explain some concepts, and under explain others.
However, we'd be grateful if you could give us some specific examples of sections or pages that you didn't find useful or enlightening.
The idea number 0 is component interfaces. They seem equivalent to plain old interfaces in all the other languages. (Except some reflection may be necessary in statically typed languages to process the lists of functions in a generic way?)
The idea number 1 is bases. They are domain-agnostic adapters that convert component interfaces (aka language-native interfaces) into language-agnostic interfaces. (Except they seemingly do include domain logic because they combine several components before exposing them?)
The idea number 2 is project layout, I think? It appears to be more of a supporting convention at best, not sure.
Question: What parts of Polylith are important and what are just “cermony”?
Answer: The short answer is that all parts are needed:
interface: Enables functionality to be replaced in projects/artifacts.
component: The way we package reusable functionality.
base: Enables a public API to be replaced in projects/artifacts.
library: Enables global reuse of functionality.
project: Enables us to pick and choose what functionality to include in the final artifact.
development: Enables us to work with all our code from one place.
workspace: Keeps the whole codebase in sync. The standardized naming and directory structure is an example of convention over configuration which enables incremental testing/builds and tooling to be built around Polylith.
I also want to add one thing, and that is that Polylith combines the LEGO-like blocks of code (components/bases) outside of the language itself. In Clojure we use tools.deps, but in other languages we would use other tools, like Maven in Java, to combine the different source directories into projects (that is built into aftifacts in the end, like libraries, tools and services).
For example, in a react / redux app, you would have "pure" components representing the business logic of your app, either a component or a base that actually stuff them in a redux store ; a multiple "react-components" that returns jsx ; and at least one "react base" (or react/redux base) that ties everything.
Different projects would be "your real application", "your dev server connected to local stub services", "your storybook execution", etc...
I try to explain it in the "Bring it all together" section also: https://polylith.gitbook.io/polylith/architecture/bring-it-a...
How exactly are these deployed? Does it use k8s, Docker? Does it integrate with some particular CI setup? How can this be language agnostic? How can a single codebase be separated into multiple running http services without some kind of overarching routing framework? How do services communicate?
Polylith doesn't help you with the deployment of the project, that code you have to write yourself, but it's possible to calculate if a project has been affected by the latest changes or not (the 'poly' tool that we have build for Clojure, supports that, but it can be implemented for other languages too, and what you get is support for incremental builds and tests = only build the project that are affected by the change and only run affected tests).
Polylith is an idea about how to decouple and organise code, in the same way that FP and OO are ideas that can be materialsed in different computer languages.
The whole codebase lives in a single repository, but all components (and bases) have their own src, test and resources directories. We then pick and choose from these sources to create projects from where we build artefacts.
All projects live in the same repository in a workspace and if you structure the code the "normal" way, each project may live in its own repository with a single src directory (+ test and resources) but then you had to put all the code into that single src directory. What Polylith does, is to let each project specify (in a configuration file) which set of src directories it needs, instead of putting everything into a single src directory. This is supported out of the box in Clojure, but in e.g. Java, if you use Maven, you need a Maven plugin for that.
The deployed services communicate with each other through their public API. No difference from what you would otherwise do.
Call me a pessimist, but I'm not convinced. It seems a bit hand-wavey to me.
Say in your Real world app you have a SQL for getting articles by user in store.clj. How do you decide if that SQL lives in article component or user component?
To answer your second question, I would make the same considerations as in any other architecture. If the SQL only took a user ID and didn't contain business logic from the user domain, then I would just put it in the article component, but if the SQL had to contain logic from the user domain (more than e.g. just translating user name to user ID) then I would extract that logic into the user component. Exactly how you combine this into a query is an implementation detail that can be solved in several different ways (and it often gets quite ugly!). This is how I think about it in general, but there are always situations when it's right to break "rules"!
If you don't mind a drive-by suggestion as I keep reading, this should be the first bullet in the list. It's a tiny speed bump in the funnel, but I could feel my attention dwindle until I got to "oh, there's more to read if I click the large, friendly button".
…Like functions and values?