A Theory of Software Architecture
danuker.go.ro
danuker.go.ro
This is what I’ve started telling people:
Use mostly functions, try to make most of them pure.
I think that can get people (even new devs) 80% of the benefits (testability, composability, loose coupling, and the ability to reason about code) of more complicated, prescriptive architectures (Hexagonal, Onion, Ports & Adapters, Clean, etc) with a minimal amount of ramp up.
Of course, this isn't the solution to every problem (it obviously depends on the domain you are working in, for me its webDev and backends), but I think maybe its a good way for people to start.
Edit: Here is a great talk demonstrating that by following a simple functional approach, your code can naturally fall into a “pit of success”: https://youtu.be/US8QG9I1XW0
This reminds me of:
> Eat food, not too much, mostly plants > > -- Michael Pollan
My new mantra: Write software, not too much, mostly functions.
Did you have something else in mind here?
The problem is that, as the project continues over time, the sweet spot moves...
If you nest your models/components deep enough that they're forming new abstraction layers, you're nesting them too deep. Backup and use mixin-style stuff instead.
If your business logic or application are nesting pretty much at all, then you haven't succeeded at making good choices in your models/components.
Maybe... Duplication is a code smell, but shit abstraction is a code problem...?
If your product or tool lacks a clear focus and goal, then the code base will reflect that, and will grow and sprawl endlessly.
From personal experience, I worked with a team years ago where product owners wanted a datetime picker or parser for an app we had built. Senior dev on team decided it would be cool if it included some lite NLP. When product owners heard from him how easy it would be to add, they were on board.
He started with a popular existing Python library. But there was a bug with one corner case that was causing problems. So he took the initiative to spend a few extra days on the story to write his own simple NLP date parser.
A couple months later, early on Jan 1st, the new feature wished our ops team Happy New Year by taking down our application.
I happened to open an issue for the library's bug on Github after learning about it from the other dev. The owner there promptly responded to share a simple workaround for the issue. But by that time we already had too much software on our hands.
I think the most important principle is to "keep it simple". I mean, even if you have absolutely no idea how to put things together, just trying to limit LoC and trying to not do anything fancy is probably going to get you into better spot than any other principle alone.
"Keep it simple" has the advantage that it in itself is quite simple. Even if you are just starting you can pretty much tell simple from complicated code. You might not yet be able to make your code simple but if you honestly try you are also equipped to judge your results in an intuitive way which is essential to improve. The only way to improve is to be dissatisfied with your product and the only way to do that is to be able to judge it at least in some way. Usually the way you are dissatisfied is going to shape the direction in which you are likely to improve.
That is not the case with other principles, which oftentimes are easy to state and sometimes easy to show on a small scale of a very simple example, but give not much guidance on how to use and combine with other principles on a large scale.
For example, principle of using patterns to structure your code (we are talking GoF, PoEAA, etc.) which is frequently taught to people is right in itself (ie. whenever possible we may want to use established ideas on how to solve particular problems) but is frequently misrepresented by adepts who try to overload the application with constructs which even if correctly implemented, might have been replaced by a simpler construct. It frequently requires a lot of experience and good judgment to tell which pattern could be used in particular situation and it absolutely gives you no hint of how much is enough giving impression to novice developers that the more patterns you cram the better developer you are.
I also think that if you put "keep it simple" as your main principle, in your search on how to "keep things simple" you are probably going to discover other principles and also understand why, in the process.
I have many times seen codebase that has been thoroughly obfuscated by "pattern wannabees". I have also seen codebases made by teams that had very little knowledge of programming (like barely being able to program their way out of paper bag). Of the two, I very much prefer code written by people who don't try to be too fancy.
It takes twice as intelligent person to read the code so if some intelligent people try to be smart with their code but are misguided, it freqeuntly results in a codebase that is very hard to untangle.
With a code that is naively written, the problems tend to be simpler both to understand and to resolve.
Couple of months ago I had a discussion with a dev who was supposed to implement circuit breaker on one of the services (dunno why on one service only) and the resulting was a dozen pages of Java code that was chock full of callbacks, suppliers, optionals and what not. But something did not sit right with me because I could not for the life of me see what the code actually is doing and given the problem it was supposed to do I did not expect so much code.
So I sat down and over two hours I have simplified entire dozen pages to a single try / catch with an if in it.
It seems like in the industry the cost of having state has always been overlooked.
Funny enough, in frontend software this pain surfaced a lot, but in the backend it keeps being unnoticed, at least in the world I live in (Ruby, Javascript).
https://en.wikipedia.org/wiki/Multitier_architecture
But I prefer the first version of the example, because while it's named "complex code" I think it's the simplest version. I think that there is no need to decouple this simple and straightforward function in three functions that cannot be reused independently. Splitting a function in smaller functions make sense where the code is complex, but I think it's important to not overly do it.
Both ways could go wrong. I don't care what your architectural choice is, you can still mess it up.
The arch proposed by the article has "Application Business Rules" (ABRs) and "Enterprise Business Rules" (EBRs). Now in a small start-up, what are the "enterprise rules", is "enterprise" not merely a buzzword in this context? And how ABRs and EBRs differ is not well explained.
In MA this is much clearer: the Data Access Layer (DAL) contains DAOs (the persistence side of the model in MVC), Business Layer (BL) contains the business/domain logic (the other side of the M in MVC and/or the business logic that may end up in the C of MVC; aka services in Rails), the Application Layer (AL) contains what would be typical "controller logic" (authorization/redirection/data gathering for the presentation) in MVC and the Presentation Layer (PL) contains the V from MVC.
> [...] I think it's important to not overly do it.
Yups. So maybe one can do without a BL at first, and put the business logic in the DAL at first and dont mind to have a litttttle bit of it creaping into the AL (controller).
// build URL
// find "definition"
Clean architecture also differs from n-tier in that–although there are layers, they don't have to be physically distinct and the APIs between them are not typically heavyweight gateways designed to work across network boundaries etc. The key design activity in CA is reducing the coupling between abstract business logic and the concrete code that executes it. It's actually quite a simple approach to describe, but more difficult in practice.
There's nothing particularly wrong with the first function; functions certainly don't only have to "do one thing" in clean architecture.
Sometimes a big, fat block of code is just easier to understand.
John Carmack made once the same comment: http://number-none.com/blow/blog/programming/2014/09/26/carm...
As a functional programming enthusiast, I agree with this.
Recently I worked on rewriting a relatively large backbone web application in React/Redux (For those that don't know, Redux is a framework for writing JS in a more functional style).
While moving all app logic to functional style makes it safer and easily testable, it is definitely not friendly to most programmers that maintain it. We've had lots of bugs caused by new people coming in, not understanding the functional style and making bad changes such:
- Doing side effects from functions that are assumed to be pure.
- Writing a lot of logic into functions that are directly doing side effects, instead of refactoring the decision making into reducers.
While I still love mostly pure functional code, I would only recommend it for smaller projects, where one or few developers with strong grasp of the style would strictly review every new PR to ensure new people don't mess up.
That completely goes away as soon as your language has purity declared at the type system.
It's just not available for Javascript development, like most nice FP features.
I had the exact same experience with people not understanding new paradigms with Procedural (programmers experienced with Assembly using GOTO for everything), OOP (programmers used to procedural using only static methods), MVC (by putting everything in the controllers and ignoring views/modules/helpers), MVVM (by modifying state by themselves instead of using the MVVM mechanism).
All those things were always "obscure" for newcomers since it was different from what they learned in college, but after a while they became second nature.
I think the answer is not to avoid those paradigms because they're hard, but rather to teach people how to work with them. It's expensive but it's only way forward IMO.
If you guys moved to a fully functional paradigm where the language enforces the functional style you will see greater benefits.
Unfortunately for the front end the ecosystem for functional languages outside of TS/JS is not that great. But an easy one to try out to actually see the benefits I recommend writing a little app with ELM. With something like ELM, the functional style is enforced by the compiler. You will see that 90% of the bugs you typically deal with in JS/TS will disappear. One type of bug that will disappear is runtime errors. Runtime errors are not possible in ELM.
I find a lot of JS programmers haven't fully grokked the functional style. Example: You'll find JS programmers who talk about how much they like functional programming but don't understand why for loops don't exist in functional programming.
> In no way, shape, or form am I making a case that avoiding function calls alone directly helps performance.
Rather, he argues that (among other things) this is a way to make current bugs more visible and to avoid future bugs by disallowing calling functions that should be inlined.
If you don't have that, at least big fat blocks of code are self documenting...
It has been a strongly held opinion of mine for a very long time, but I haven't been able to state it as eloquently as that.
fn x() {
doStuff();
moreStuff();
forgotSomething();
}
fn x(y) {
foo = doStuff(y);
bar = moreStuff(foo);
if (isSomething(bar)) {
return theRest(bar)
} else {
return theBest(bar);
}
}
It's important, however, that these helper functions are not haphazardly strewn around the code base and accessible to things that don't need them. Depending on the language/context, I'd reach for nested function definitions or public/private keywords (or a combination), because definitely, it can be very hard to approach a big file with a bunch of (often poorly-named) functions that are defined at the same hierarchical level but not meant to be used at the same level.
It seems fairly obvious to me that either the former example is waaaay simpler or it is doing something really distasteful.
There's not much else for them to do, since they don't take arguments, but to muck about with global state and/or have side effects.
Given that, I have to say that I, err, and I'm sorry if this sounds a little arrogant, but I don't really believe you when you say that it's easier to understand. It might be easier on the eyes, sure, but to really understand how the first one works, you'd have to look into all the defined helper functions, and see which outer-scope variables they touch (global/module/class scope), and also, probably, have some knowledge of where x is called. In a style closer to the second example, there are less outside dependencies.
Of course, this is all very abstract. I'm not saying there aren't cases where a few variables in global (or class) scope that get mutated by bare functions can't work at all, but as a rule I don't consider it a good style.
Even just seeing a few variables or objects passed around, I still don't know exactly what it's doing, or hiding. I can more quickly understand the high level idea with the former example without the details potentially confusing me.
I think it depends on the type of code and how you interact with it. A library that requires its consumer to keep track of a bunch of data and pass it in and out at the right times when it could just as well handle internally is not a very good library. But if we say that all the code the above example is from an application, where a normal change might involve touching all of the mentioned functions, making it explicit where some variable is used.
It's independent of OOP vs. functional, through it's easier to have implicit arguments and mutation in OOP so you see it more often there, especially if people somehow ended up believing that programming OOP means you need to create a class for everything and make all states a object member instead of e.g. a variable in the stack.
...unless we take a more compositional approach and define our application to consist of many "parts" each of which has a public API and an implementation. Well, in that case I would argue that each function can be considered as such a part, having a public API (the definition of its inputs and outputs) and an implementation that can be changed while keeping the public API stable.
Over the time I came reached a conclusion: whenever I have to run some code to be sure it does what it should, I write a test instead. And every often that is a unit test. If I'm highly confident that the code will do what I expect, then I rely on very high level system tests, unless I fear that someone might break the code later by accident without the compiler (or other existing tests) being able to detect that.
I want code to be "clean and simple". Thats how I judge my own code and everyone else's. And the first example is indeed cleaner and simpler.
In my theory, you are only allowed to introduce more complexity (functions, frameworks, etc), when it makes it overall simpler and cleaner than before.
def find_definition(word): # Listing 3
url = build_url(word)
data = requests.get(url).json() # I/O
return pluck_definition(data)
Now you have a 'build_url' method, which is a bad name to start with. It's actually a 'build_def_request_url' or something like that.
Imagine you add another method find_synonyms(word). My quick brain would think I can use that same build_url() method. Wrong, because one needs a build_def_request_url() and the other a build_synonyms_request_url().
Let's say I need to debug and need to see what url it's calling. In the original code it's right there, here I have to on a goose chase to find my answer. One thing that wasn't refactored is putting the hardcoded url into a constant. That one I would do.
Another complexity is adapting one of the helper functions. If I adapt it, which callers could I break? Who is using this function?
Another part is that I would rewrite the pluck_definition(data) to a more generic get_value(data, u'Definition'). That way my other potential find_synonyms() could also use it, and it basically adds no extra lines of code except for an extra parameter.
I know this is an easy example to explain things, but the original code needs to be way more complex to justify it splitting up into multiple functions, that in this case can serve nothing else but that original function.
Just my feeling towards this code.
Whats important is to not have leaky abstractions though. The moment you need to start reading into those functions is the moment you realize you would have been better off leaving the code as is.
Its very hard (at least for me) to come up with those layers and requires a ton of iteration. Thats why I also prefer the top example - you need to work with the lower level code first, and go through enough of it before you can start spotting patterns and creating layers, otherwise you’re bound to mess things quite a bit.
Tradtional mulit-tier the UI depends on Business Logic, Buinsess Logic depends on persistence etc
The idea is protect the core, and persistence or UI concerns adapt to how the core wants to interact with the world.
Others find it tiresome to read every line of code and develop intuition around abstractions instead. The "find_definition" function will be all that is parsed.
The funny thing is people shift between the two as the scope of the project increases.
He uses such a simple example so that his idea that applies to much larger, more complex codebases can expressed in a presentation/article, not to argue that all such simple functions need this level of decoupling.
Coupling kills software
I hear this a lot but I think it’s highly overstated.
The “clear” final version is still strongly coupled -- you can’t call find_definition without it directly calling the build_url helper! The key aspect isn’t the coupling, it’s that the high-level imperative function is calling a simple, low-level, testable helper.
Coupling can be bad, sure, but not if it’s kept under control. And excessive decoupling can make programs much harder to understand and debug! Have you ever worked on an app where every reference to another object is injected via a DI framework, and where all the significant calls that actually do stuff are farmed out to asynchronous messages and callbacks, all in the name of decoupling and testability? That can make it really hard to debug problems. A good balance is what’s needed, not decoupling over all other concerns.
And it's injected with an interface, not a concrete class, so you have to go on a giant easter egg hunt to even figure out which class is being injected... only to eventually discover that there's only one class that implements that interface.
This is not a joke. Spring invalidates refactoring tools and a lot of static analysis. It makes working on Java feel like you are working on Python or something. The traceability of errors is very low. You can limit yourself to a subset of Spring's functionality (eg no component scanning, wiring up the beans manually) but it will not be idiomatic.
I'm dealing with a project at the moment where they haven't done DI and everything is dependant on concret classes and I just can't write tests (without huge setup methods) for anything non-trivial and I have these large constructors initialising things in every class.
From another POV it's IDE vendor lock-in: make the basic programming exercise so challenging that the users have to buy your special tool to even attempt it.
If they had accepted the evidence and backed off the tool until it could deliver the better than C quality (which it did do) at the same speed of development they might still be around. However in the end we need to ship code to make money and our competitors were catching up on us which isn't acceptable to the business.
Let this be a lesson when you want to adopt new tools. They might be better but they might have some unexpected downside that means they are not useful.
I mean, in this case it's literally:
* right click on "someInjectedService"
* select "Go To Implementations"
and you'll be there.
Generally I do relate to the complaint though. DI makes your project feel like a "system" as opposed to a "program" (not that this is a bad thing or unnecessary).
> I hear this a lot but I think it’s highly overstated.
Coupling is like trying to get to your destination by swimming through molasses – each move you make impacts atoms far away from you because they are tightly bound to the ones near you, and those atoms exert a force back.
The real thing you should be thinking of is, look at those libraries and their documentation / READMEs (depending on library size, smaller actually preferred). Can I make my code modules more like that? Will the API boundaries I define make sense?
The second mantra is to think a thousand times before you name something. I actually keep a list of names (..Manager, ..aggregator,etc.) gleaned from various sources, to name my classes. Earlier, I used to name most of my classes as <Function/Operation>Manager
Edit : Here is a question, that actually prompted me to keep a list.
https://stackoverflow.com/questions/1866794/naming-classes-h...
I will adopt the list idea!!!
Names ending in Manager are usually a poor choice.
See Peter Coad's "-er-er" principle:
> The “-er-er” principle. Challenge any class name that ends in “-er.” If it has no parts, change the name of the class to what each object is managing. If it has parts, put as much work in the parts that the parts know enough to do themselves.
See also: http://www.carlopescio.com/2011/04/your-coding-conventions-a...
Peter Coad advocated against this in favor of modeling the problem domain under consideration using an object oriented approach.
In an object oriented domain model there is no place for such external “controllers”, but I think the commenter doesn’t propose this approach.
Really curious. Do you have any material that explains this way of design?
I work on mostly web apps. End of the day, it's really about moving data and transforming data. So most of my programs have no choice but to deal with data, and so, my whole design process revolves around gathering, storing, operating upon and transferring data.
Why would a TransportMasterManager, which provides factory methods, enumerations, a collection of the current set of TM's, and knowledge of the current user choice for which TM's is being used for what, have "no place" in this design?
> AppDomainAffine, therefore, would be a more appropriate name. Unusual perhaps, but that's because of the common drift toward the mechanics of things and away from concepts (because the mechanics are usually much easier to get for techies)
how is 'AppDomainAffine' not a concept?
Naming things: I try to not think about it for more than 10 seconds, and go with the best I've got by then. I find myself renaming things sometimes, and I'm eager to do this when a better name comes to me.
Just name it whatever and come back to it when it starts to matter more and you've probably thought of something better by then.
At that point, it is a cognitive burden to handle so many "managers".
https://stackoverflow.com/questions/1866794/naming-classes-h...
Distributor is a better one sometimes. For example, ConnectionDistributor instead of ConnectionManager if it accepts connections and distributes them to a thread pool.
I gave up .... I write mostly in languages that allow you to deterministically rename things with refactoring tools, so I frequently rename important classes 5 or 6 times before I'm done.
Not only renaming, but re-bundling entities, moving layers, changing abstractions – refactoring is crucial during and immediately after development. As you implement your idea, you will inevitably find a better way to express it, and it's crucial to be able to re-do these things as many times as possible to reach the best possible result (if it's not a throwaway prototype, of course): no maintainer, including yourself a month later, will have a picture as full and clear as you right after finishing the first iteration.
https://steve-yegge.blogspot.com/2006/03/execution-in-kingdo...
If one's position is that striving for a distributed model in memory is the wrong approach because of the inherent complexity associated with distributed systems that's fine but I rarely hear this argument. Typically, OOP/OOD is criticized based off of patterns that aren't proper.
https://www.infoq.com/presentations/Making-Roles-Explicit-Ud...
The problem with OOP is that no method is truly pure, no method is a combinator.
class Thing
var memberVar
def addOne():
return memberVar+1;
You can use static functions but the static keyword defeats the purpose of a class and makes the class equivalent to a namespace containing functions.
class Thing
static def add(x):
return x + 1;
namespace Thing:
def add(x):
return x + 1;
Use namespaces and functions when all you have are pure functions and use classes when you need internal state... there is literally no point for OOP if you aren't using internal state with your classes.
For me, the challenge has never been with the architecture or our combined technical knowledge.
What I have observed as the main challenge is that most technologists start solving the problem before they know what the problem is.
You could argue that you should talk more with the stakeholders, but most of them don't have the skills required to accurately identify their own problems. Instead they need to see something running and then notice when things are missing, which is why we start building stuff without knowing what problem we ultimately are supposed to solve.
Unless you know your goal you will never be able to deliver on your customer's expectations.
There's lots of software in the world (all creative software, for example) that doesn't share these attributes, and all the skills of a BA are useless.
The scope of software is larger than webdev and larger than business.
I whimsically develop programs for myself that I could manually sort out in 5-10 minuts if done sequentially.
Another equally large challenge is that the person/people with the problem to be solved doesn't actually understand the problem and is incapable of describing it in a way that makes it a priori understandable.
In the example, there is hardly any code that does anything different depending on state. There's no state being managed, so there isn't actually any architectural problem being solved here. Should the API go down or change its format, the code breaks. The pure pluck_definition() will still fail to parse the JSON if the format changes. The pure build_url() will stop working if the API changes its URL format. They will pass unit tests, but fail in practice.
An actual problem to be solved here is to abstract away the details of the REST API, formatting and network errors. One way to do this is to pack that into a component with a well defined interface. You can still do this stateful/non-stateful split within the component if you want, but on the application level you need to apply that heuristic recursively at different levels of abstraction.
Why is this so bad? Its not because its expensive, yes that is bad, but the largest issue with working in a ball of mud architecture, is that the code becomes so fragile and interdependent that changing any one thing can easily lead to breaking many other things. This leads to a culture of fear of change which grows tech debt. Then one day someone steps up and decides to actually refactor this ball of mud to have some semblance of logic to it, what a noble soul. That person is then subject to a barrage of bugs and issues from the refactor and is that the mercy of their supervisor.
Dealing with state and other side effect like issues is certainly something to consider in architecture, but it is a different argument entirely.
The pain is the title..
People pick catchy titles; it's a key part of getting noticed these days.
Keep in mind, Ruby isn't a functional language, but here was this presenter describing essentially how you write good Haskell code, but in Ruby! A language that makes it so easy to mutate in place that it's known for libraries that "monkey patch" base classes and can change the definition of operators and functions at runtime!
Fantastic talk, fantastic presentation. I now share it with my new engineering colleagues too.
Here's the talk where he talks about imperative shell, functional core, "Boundaries": https://www.youtube.com/watch?v=yTkzNHF6rMs
I'm not aware of any good OSS examples, either.
I think React and Redux are in a way an implementation of these concepts. React is a view layer that sends messages and each component is (ideally) a pure function of state, and sometimes its own history. It's so, so easy to unit test the interactions.
Hooks make it a little harder to reason about, and it's unfortunately easy to write hooks that perform IO and you suddenly end up in a miasma of difficult-to-test code where you have to return to using a mocking library to "replace" IO functions with fake versions, and then you're on a slippery slope again toward mixing interaction and mutation in one layer.
IMO it's pleasant to use Ruby in a functional style, much more pleasant than Python owing to the ease of chaining calls that take lambda arguments (blocks). Every time I try to do something similar to a big chained Enumerator stream in Ruby in Python using a list comprehension, I have to invert my thinking - as far as I'm concerned, those things are written backwards and inside out.
There is some faucet, transformation and then a sink. Only at the sink does external state get mutated.
It helps because transformations can be closer to pure functions and you know there is no state changes until you've hit the sink/loader.
Not sure yet, I haven't concluded it's the right way for me, as it has saved a few headaches here and there - but I'm sure I've unknowingly caused more else where just yet unseen.
Also for the author, my favorite word of idempotent - given the same inputs - always get the same output.
For example, boolean negation is a pure function as it depends only its input and has no side-effects, but it's not idempotent since: not(not(var)) != not(var)
In this case, the first listing is simple and easily readable.
The introduction to the article relies on so much context and assumed knowledge that I almost didn't read past the third paragraph.
Who are Uncle Bob, Gary Bernhardt, and Mr. Brandon Rhodes? I have no idea, and I don't need to know who they are to read the rest of the article. (Which itself is extremely well-written.)
IMO: Have an introduction that doesn't rely on unneeded context. It's appropriate to credit people; but do it in a way where a reader unfamiliar with the context doesn't assume that they need to know the context to read further.
It is important too attribute ideas to their originators (or at least the source from which one has learned them) and it is good that the OP is doing that.
I agree it could have been done a little more gracefully, though, with just a minor tweak to use phrasing that doesn't seem to assume the reader is familiar with those dudes.
Testing pluck_definition by itself is completely pointless since it does nothing on its own. This is the “test public interfaces, not private implementation” principle.
Similarly build_url and pluck_definition need to be coupled because they both depend on the specifics of a third party API. It makes it much more clear what the expected output is to keep them together so that if something breaks you know which url to check and what the response should look like.
I also dispute that it’s hard to unit test the first function—-you would simply mock the api response and then you have a great test. Much better than having separate tests for tiny helper functions that do nothing on their own.
Now maybe this example is just too simple and the presented architecture makes more sense on larger code but if so then the article is poorly written. Examples need to be realistic enough not to obfuscate.
How many times have you seen a test pass or fail when it shouldn’t have because the mocks didn’t match the actual code? It’s a useful technique but it has drawbacks which are not easily prevented.
If you think about the problem more, ask yourself how it's possible to mock a remote HTTP call without the risk of this problem? I've seen people write frameworks which periodically save and cache responses but unless you regularly perform some kind of validation there's no way to avoid that problem.
I simply don't find the article to provide a grand unified theory across different genres of software at all. What worries me is that newcomers to the industry will, until they move to the next discovered silver bullet unified theory.
This is the first thing you need if you want to meet any business requirement at all.
While this is possible to do, it is easier said than done. Organizing the code this way requires a lot of time dedicated to thinking/rewriting, which is often not available in the usual corporate environment with strict deadlines.
Eg. I can't imagine what would be hard for a MS Paint clone to achieve using this approach. There are always things with side-effects (IO, namely), agreed, but in this CRUD example, the integration bits are coupled in a way where you need a single trivial integration test. Similar approach can be applied to a GUI app, so I am not sure why does GP feel it can't? Any concrete things where you think you can't apply it?
Sure, you'll have many more small functions, but I think the core takeaway should be that you can always structure code in away where integration functions are simply statement-lists (eg. no control-flow logic in them). That will usually require rest of your code to be functional.
And yeah, it's harder to adapt existing code to this pattern, but introducing new code in an existing code base following functional pattern is trivial.
I'd say that you can apply it mostly everywhere, however it is not friendly to most people that will maintain the code, and that is a problem since most software will have many maintainers over its lifetime.
This reply to another comment I made on this thread should elaborate a bit more: https://news.ycombinator.com/item?id=24917764
While the current education system is geared toward procedural programming for the most part (I imagine mostly theoretical computer science curriculums only focus on functional programming and lambda calculus too, but even then, only very late and very theoretical), the question is more of whether it is a better approach when applied "universally" (with non-functional languages, it's unlikely to be really pure)?
If deemed that it is, functional proponents like me (and you, it sounds like) should push for it to get a better coverage in Universities than eg. OOP, even for OOP languages. Most academia is out of the software industry, so should we educate them or not? And how best to do that if the answer is yes?
I do have a worry that some of it is also incomprehensible to some people, or that the barrier to entry is higher. Is such purity more reserved for those that also like mathematical abstractions?
Now, the biggest problem I have with colleagues reviewing my code is that it seems too-simple, and they would have introduced another 2 layers of indirection/abstraction, but they can't really say that anything is wrong with my approach. It's really hard to get them to jump out of their "OOP bubble".
The code is easy to maintain, but there is a big risk that someone will pop in and just turn it into one big side-effect mess that will be hard to maintain. But then again, that's what they would have done anyway, this has at least some chance of not becoming that :)
I wish there was a talk, that took something like Microsoft Word and de-constructed it and explained how someone can program it, from first principles.
The Word codebase is no work of art, I guarantee. Like any codebase of any size, it's more of a ball of duct tape and bailing wire.
A purely functional approach would involve a core of functions taking in a current state and returning an updated state and a list of canvas actions to take.
Functions might include "select ellipse tool" and "set pen width to 5" and "click on canvas at point (3,15)" and return canvas primitives like "render ellipsis from (3,15) to (47,99) with stroke width 5, stroke colour black, and no fill".
Unit testing of the core should obviously be quite simple to do, being totally independent of the GUI system.
I simply don't see how this fits the functional core + imperative shell pattern.
while not is_closed(state) {
(mouse_activity, kbd_activity) := get_mouse_and_kbd() //blocking call
(state, is_modified) := update(state, mouse_activity, kbd_activity)
if (is_modified) {
draw(state)
}
}
The one issue here is that once your state gets to a certain level of complexity, as it certainly does in ms paint, it's not going to be performant without immutable "persistent" data structures, which are either not available or not widely used in python/js/ruby/c.
If you're thinking i haven't achieved the goal here because the draw procedure is big and complicated and imperative, then maybe we need to replace
draw(state)
view := render(state)
draw(view)
From a explanatory pov: I suspect you're thinking that the mutable text state should be "core" but it's actually not, it's part of the "shell". Think about a RESTful frontend providing UI to a (mutable) database holding the state, the database conceptually is the "core" of the program, but in parlance of the architectures discussed here it would be in the "shell".
It is a bit confusing, unfortunately, and it's one of the things I dislike about this classification scheme.
A MS Paint clone is one of those apps that fits rather nicely on a Clean Architecture.
You have the domain model (raster image), you have the application layer (image processing operations over the raster image), and you have the IO/service layer (image exporters/importers, GUI, platform-specific features, etc).
You sound more like an academic with no real life experience on large software.
You do realize that within those "neatly separated layers of yours" there will be an insane amount of complexity which will be impossible to separate in layers?
Why, yes I am.
> You sound more like an academic with no real life experience on large software.
Well, I do work as a software engineer for a FANG, so that does fit the bill I guess.
> You do realize that within those "neatly separated layers of yours" there will be an insane amount of complexity which will be impossible to separate in layers?
Well, there really isn't you know?
I mean, unless you don't know what you're doing.
In my line of work, the hardest problem is to reign in complexity. It's very easy to mess up and end up with insane amount of complexity, specially if you don't have a clue about what you're doing.
In fact, as the saying goes, incompetent developers make complex things, while competent developers make simple things.
It seems you make a living struggling with complex things that ends up piling on a lot of complexity. Perhaps you would do yourself a favor by improving your skillset and learn how not to follow that path.
You on the other hand still belive you can make everything totaly managable forever and always and in any situation.
Your academic tone betrays you: you obviously believe you can always split things into neat little parts and then write books about it and how even in production this is so easy and always "just a bit of layering and it will work".
Anyone who worked on real production systems over years knows that unless your company has infinite money for developers and infinite time that you will sooner or later get parts on which complexity starts getting layered on.
But hey, "just add a few layers".
Also being a FANG developer means nothing. I've heard stories about FANG company developers, they are not very FANG-like to say the least.
This, to me, feels very very similar to a model-based approach, where the outermost level is a modeling framework that does nothing more than route data between different functional components and IO components.
I have a strong hunch that this outermost layer does not need to consist of anything more than a suitable framework plus some configuration data specifying (a) the mapping from functional components to hardware resources and (b) the data flow between functional components.
If this is indeed the case, then, extracting individual components or subsets of the system for testing should simply be a matter of providing a suitable transformation of the configuration data, and re-executing the framework.
I've found these organizing principles to be quite useful.
1. identify the key that your application is processing (in this case, it's `word`)
2. make it so that as many functions as possible take this "key" as their only argument.
3. whenever you are fetching data about a model, it is either directly or transitively in terms of this key.
4. push model-fetching (technically the I part of I/O) as far to the leaves of your program-tree as possible, hiding them behind "model client" classes, that are passed in to your job at construction.
5. Perform all upserts idempotently, atomically, and in the bottom-right corner of the execution tree. This makes it easy to reason about mutations, and also easy to omit when you're wanting to do "dry runs" or read-only local runs.
6. (4) means you will occasionally fetch the exact the same model more than once in the scope of one execution. This is OK. You can optimize this later with execution-tree-scoped caching.
With this approach, the user-directed I/O (imperative shell) is at the root of your tree, and very narrowly-defined to be the key of computation. You can build a run-loop on top of this, or a CLI tool, or an rpc service. It's narrowness is kind to all kinds of interfaces.
The functional core is everything that isn't the leaves of the execution tree. These functions essentially compose model client calls, and only take the keys as inputs.
The leaves are a collection of reusable one-to-five-liner RPC requests / data base queries. If you like, you can wrap these in a class that caches the queries, as described in (5).
The tests for leaves mock nothing if they target a DB. They mock the rpc service if they target an RPC service.
The tests for the functional core functions mock the model-clients (leave functions).
With this structure, as long as your tests are brittle enough to break when an RPC contract or data model changes, you do not need integration tests.
I do this in Python, but it would work just as well in any language with structs/interfaces. I've observed the same benefits proclaimed in the article, although my approach is a bit different.
If people are interested, I could write more on this, with examples. Let me know!
It's true though that software architect is a vague term in our unregulated industry. Many companies never have software complex enough to need an architect; and many struggle through lack of a global plan, and just have lots of local decision making and get slower and slower over time.
However, the functional part of the discussion, while I agree with its benefits, is highly depending on the domain within the onion.
There are idiomatic ways to program in particular languages (even in Python or JavaScript) which are strictly against the functional approach, even though nothing in those languages prohibits it. It gets trickier with external dependencies which are "forced" on you too.
Do you have a concrete "domain" example that you think will be hard to turn functional (which basically means turn the integration points into minimal functions doing just the integration bits — basically, any side-effects are limited to those integration functions)?
You can easily turn things into substructures and have functions only work on those: whether you pass by reference or value is up to you and your choice of language, but even when passing by reference (to avoid memory copying), you can write functional code.
Again, it sure is non-idiomatic for most languages, but that does not mean it's impossible or even hard.
As far as "absolutism", generally, aiming for minimal statefulness will help you write more maintainable code (citation missing :).
Now, there is a place for pure functions in that environment. But making decisions based on state can run so thoroughly through the code that "functional core" leaves very little core.
My initial hunch is that you've got a lot of existing code that passes and mutates big global state objects around. Still, even your description clearly highlights an issue that is there regardless of whether you want to push for a "functional core": "decisions based on state ... run so thoroughly through the code" _will_ come back and bite you.
It also highlights an easy way to decouple those into simpler functional parts: identify which part of the state is really needed in each part, and only pass that in, and have it return an updated state: basically, the only change you are making is turning the implicit parameters into explicit function arguments and return values. Turning the entire codebase around will be tricky, of course. But perhaps you can switch over chunks of it as the units are readied as you go along.
Of course, if you've got a lot of data that would be expensive to copy around, you might want to keep some of the logic for mutating that non-functional, but you could still decouple that thus making it have a functional core too.
You have 6 or so different sources of control (different panel systems, automation systems that use serial interfaces, other automation that uses Ethernet). Each of those is a different thread.
All those different sources of control need to see the same image of what's connected to what. So when one thread makes a change, it has to change for all the threads.
You could think about separating that state into parts, but does that really gain you anything? If you've got 80 variables that behave identically instead of one 80-element array, are you really ahead?
Another (equivalent) way to test it is to mock out requests.get and response.json using a mocking library: instead of performing real requests, do what the test wants (return correct data, return unexpected date, or throw an exception).
You also get the option of putting complex operations in plpgsql functions when you want them executed atomically without any side effects, corruption, or cleanup requirements when one of the step fails. Your default failure mode becomes "You got an error, everything was automatically reverted, you can safely tweak your request and try again".
Also operations being performed in a monolitic db means that the different pieces of data are much more likely to be loaded in adjacent memory caches and ready to be joined, combined and processed, providing significant efficiency and speed gains.
The main motivation for this, is because I use what I call “evolutionary design.” I tend to refine design as I progress through development, as opposed to having it substantially complete at the start (A lot of classic developers probably just defecated masonry at the very thought of that, but it WFM). Having a finer granularity goes a long way, in supporting this methodology.
It also helps a lot for refactoring, improvements, and testing. The overall quality of my products is drastically enhanced by modularity.
What you can is make is define a starting point app in the desired architecture. Then create a list of functional changes to go from starting point, to desired state.
Then you can use various stats for each architecture on how complex was the changes.
Make the functional starting point, the functional list of changes and stats a standard.
It’s possible to build a DDD Repository in Django or Rails, but really a lot of work. I think the frameworks like sqlalchemy and NHibernate seem to do things a bit better by tracking “dirty” models, hiding the “ORM-ness”, and letting the higher levels control when to flush/write to the DB. But the more you abstract this stuff, the more you lose the auto generated sugar that makes frameworks like Django so productive.
[Uncle Bob is clear](https://blog.cleancoder.com/uncle-bob/2012/08/13/the-clean-a...) that the inside circles must not know about the outside world.
That is, your business logic is not allowed to know what a DB operation is, nor should it do any DB operations behind the scenes.
> Frameworks and Drivers.
> The outermost layer is generally composed of frameworks and tools such as the Database, the Web Framework, etc. Generally you don’t write much code in this layer other than glue code that communicates to the next circle inwards.
> This layer is where all the details go. The Web is a detail. The database is a detail. We keep these things on the outside where they can do little harm.
This discussion has gone on for a long time, e.g. see the back-and-forth around https://dhh.dk/2014/test-induced-design-damage.html after Weirich demonstrated what sort of thing is required to use the Hexagonal Architecture with Rails.
I believe that if your system is sufficiently complex, you'll start to see the benefits of a more structured architecture, but it's a net drag on productivity for small projects. These days perhaps you chop your monolith into microservices before reaching the ROI point for Clean/Hexagonal?
I could believe that a "framework-first" architecture is more productive while you have <100kloc. Or maybe it's <10kloc, I don't know. I'm certainly seeing the architectural strain with a Django-first architecture in the current 150kloc monolith I'm working on, and chopping into a few services makes sense there for other reasons, so it might obviate the need for a bigger refactor onto Clean/Hexagonal.
The article discusses architecture in the abstract, which sounds more like a programming model or paradigm (OOP, functional, actor model, imperative), or maybe a pattern.
Wouldn't it be weird to discuss building architecture in the abstract, without a notion of the site to build on or the purpose of the building?
https://softwareengineering.stackexchange.com/questions/4160...
The premise is the imperative shell can become pretty riddled with decisions if your application contains a lot of I/O.
Some like the "Free Monad" solution, but I found that too be too coupling.
My favorites in regard to the topic of module structure are David Parnas and Juval Lowy
It has similar conclusions, but does it from the frame of the "story of the code" rather than abstractions like "functional core" vs "imperative shell"
Functions do work. That doesn't mean they have to have side-effects; if they're functional, they do work on the input and produce output. Doing is a verb. It's natural.
Using a noun as a function name is at best justified when you have a situation where you want to hide whether data is being calculated on demand, or fetched from some storage or lookup table. Some languages bake this in, in the form of properties - attributes of a structure which look like fields, but are actually functions. Those things have nouns as names.
1. only by reading the name of the function you know it's a pure function
2. a call to a pure function in an expression reads more naturally since operands are values and values are nouns.
Anyway, here is my naming strategy:
1. Boolean pure function
Use an adjective phrase where the adjective is the last word, for instance UrlValid or DefinitionFound.
2. Non-boolean pure function
Use a noun phrase where the noun is the last word, for instance CurrentDefinition or DefinitionUrl.
3. Non-pure "function"
Use a verb phrase where the verb is the first word, for instance PrintError or ReadInput.
That isn't IO, that's encoding. Don't you want to verify you are receiving json (whatever interface you are assuming is available) in pluck_definition? How far down do you want to go with this?
It ends up with a lot of typing but I think it is worth the trouble. Down the road it is easier to maintain and it already saves you time anyway.
You want actual theory? Here's theory:
Things like this shouldn't be titled with "Theory." Call it your "opinion" or a "design pattern." Theory is associated with things like the theory of gravity (science) or game theory (math). This title is wildly inappropriate and shows a lack of understanding of what constitutes a theory.
The fact that this post is voted up shows how much the public misunderstands "theory" and how they mistake things like this which is actually just someone's qualitative opinion with the integrity of an actual theory.
Using big words and drawing diagrams does not lend any formal legitimacy to your opinion.
It's so simple to say "look how awesome this approach is" on a 50 line program, "just use pure functions everywhere".
In real life things get very complex because there are 200 working parts interconnected. And not because "it's bad design". But because that is the requirement. Soon you get pure functions with tons of parameters or parameters that are complicated classes/structs themselves because the work that needs to be performed is very complicated.
Then you get to the issue of "this function gets the entire class as the input param but it only access a small amount of members in that class".
"Yo bro just split the class into multiple smaller ones". Sorry bro can't do, those separate classes will need to be acccessed as a whole sooner or later in another part.
People act as if there will ever be a thing such as a "perfect programming design". There won't because things will always evolve & change. Real life programs are simply too complex.
My hunch is that moving pure-functions and pure-objects out of the spaghetti will gradually eat away at the spaghetti.
These sorts of things are always better discussed in terms of specific cases instead of generalities, but I think one of the key parts of good software engineering is factoring the requirements into the simplest design possible.
And even when time or other considerations prohibit creating an extensive design ahead of time, practicing a few well-chosen heuristics like "use composition", "keep functions and methods short", "use clearly defined and consistent terminology in your abstractions" can go a long way to making code less unruly and easier and cheaper to refactor down the road.