Software Engineering principles to make teams better
principles.dev
principles.dev
If we’re talking engineering principles, not team dynamics, then it’s:
1. Use immutability by default, even in languages that make that harder than it should be
2. Understand how liberating idempotency is
3. Divide and conquer, use abstraction to make hard problems manageable
4. Delete code, delete tests, re-write stuff, re-write it again. Painful? Keep doing this till you get to the other side and feel liberation and empowerment, took me years to get past wincing at the idea of re-writing that thing AGAIN
5. Pay attention to your build tooling - to do any given task, it should always take exactly 1 command. How do i run the tests? Run the test command. How do i deploy to non-prod? Run the deploy command. Etc
1. Don’t block collaboration - e.g. if there are 5 people don’t work on 5 tasks concurrently. Work on 1, maybe 2 in some cases
2. Accomodate 2 types of work: work that needs collaboration & synchronisation (brain storming, planning, why are we even doing this, that kinda thing) work that needs focus time - sometimes that’s focus as an individual, sometimes that’s a mob focussed on one task and saying no to every other distraction in the world no matter how “urgent”
3. Change what you’re doing throughout the day, don’t try to code all day, you’ll get in a rut, you won’t write your best code after a while. You can optimise this step, e.g. if the team is low energy after lunchtime, use that to your advantage and schedule the easy boring work then, don’t be afraid to use humour or burn low energy time just getting to know each other better
4. Make sure people are being heard in the team. Hardest point to get right.It's not really about better or worse, what I've found is different people have different backgrounds and what principles they find useful is based on several things, which may be immuntable - such as strongly held values.
You're never going to make me not care about aesthetics for example, as that is intrinsic to me and that will affect the principles I like and ultimately the people that I work best with.
An analogy I like to think about is that people with very different principles are like two people holding a rope and pulling away from each other - you aren't going to get anywhere fast.
Whereas people with similish principles, will generally go in the right direction. Sure, they'll get tangled up from time to time and you won't always want to get in exactly the same way. But there's a collaborative nature to it and you'll both improve.
That sounds wrong. As in "I could upload quotes from Gang Of Four books and claim I am the source" wrong.
You can't be an author if you aren't the author of a principle or the principle is too generic. If the principle is already open source (e.g. on wikipedia, has a creative commons license) you can submit it but not claim you are the author for it and submit it under the same licensing terms (CC-BY-SA) as long as it doesn't break the license.
Codifying the principle for the first time takes effort and people can iterate on it to make it better over time. Many people may have had similar thoughts before, but if it's not a general principle already being used the first to turn it into a principle - to put a stake in the ground - benefits everyone and can help improve everyone's capability.
I believe the author should be rewarded for that effort, as long as it is their own unique work.
Then being able to create your own lists for unique situations. Say "Lupire's CTO list" or "Lupire's management principles" and to share that with your team or as a reminder to yourself.
Of course you should always be able to export it and put it in a format that's useful to you. And that's been really important to the design. From using markdown format to embedding license information and meta data with the principle, it should help a lot with portability.
but what's the point then? why spend effort for it, if it isn't expected to be better?
Learning. The result may not be meaningfully better, but you gain fuller understanding of the problem domain. The third rewrite might have a chance of being an improvement ;).
Somewhat relatedly, I've refactored processes down from 25 hours to 20 minutes. It got to 20 minutes, and other people started nitpicking that it could 'be faster' (the same people who'd let it get to 25 hours, then threw up their hands and said "it can't be fixed"). They spent loads of time getting it from 20 minutes down to... 17-18 minutes (spending 2+ weeks in the process). I'd indicated "there's really no more 'there' there - this is about as fast as it's going to get, without faster hardware". They had to find out for themselves there wasn't much more juice left in the tank.
Maybe it's not related, but I felt like sharing it anyway... :)
20 to 17 minutes is a 15% time savings still.
In all seriousness, from a business standpoint, it's almost never worth rewriting anything unless you're already making significant changes to that code. The opportunity cost of not spending that time building things that get you paid is just too high—not to mention all the new bugs that you're likely to introduce.
In my experience, most of the insights cluster in two places: just as you start writing your solution, and just as you think you're done with it. So you want to get through the whole process quickly, just to learn the things you need to write a proper solution.
Related, from a different HN thread - https://news.ycombinator.com/item?id=27692710 - "action produces information".
Throwing away a whole repository that was tested and running in production or on some dev environment, is something different in my point of view. And I think that's what people mean by "rewrite".
that's not really what a rewrite is. A rewrite is when something has already been in production, and the engineer re-do the same feature set, but refactored or rewritten in a different (possibly better, but not guaranteed) form. This rewrite takes up the time that would've been spent on something else.
If you were doing a "rewrite" during implementation, but still delivered on time, that's not a rewrite - that's just good engineering, doing exploratory coding! If you couldn't deliver on time because of the rewrite, then i guess you can call it a rewrite.
You are also assuming one just rewrites stuff blindly and randomly without any prior new insight.
That and referential transparency are huge wins. It is a shame languages like Python make it such a challenge.
For non-primitive members, `frozenset` is in the standard library, but frozen dictionaries are unfortunately lacking.
(Specifically avoid namedtuples unless you're hoisting a tuple to a real record. Record types also being a weird union with a tuple is...oft confusing)
For immutable basic data types, adopt type annotations and prefer to use sequence/iterable and mapping to direct didn't/list types. These provide statically verifiable immutability and also allow begin to encourage useful async-friendly patterns.
[1]: https://www.python.org/dev/peps/pep-0591/Would Thursday be a good time to checkpoint on this?
Thank you so much.
So now I'm more confused than when I initially read your comment. Do you mean "referential transparency" as in "expressions without side effects" or do you mean it in some other way that I don't understand yet?
Python code (pre-v3) that branches cannot be referentially transparent by definition because runtime context (the state of True/False bindings) is a hidden input in every boolean expression. You could have millions of lines of side effect free code and it will break completely if that one statement is run before the rest of the code.
Programmers depend on the referential transparency of keywords like "true", "false", "for", etc whether they're writing pure functional code or imperative spaghetti messes.
For example, let's assign a constant variable to a sequence; should you do use list or tuple (FOO = [1,2,3] or FOO = (1,2,3) )? It doesn't matter, the constant capitalization discourages you to mutate it.
It's strange that a general idea expressing (assuming?) overarching maturity has led to quite so many petulant arguments, wouldn't you say @ketozhang?
Can you give some examples?
Reading a variable shared between threads without a lock is side-effect free but not idempotent.
Similarly, many systems involve consuming from some message queue. It's basically impossible to guarantee exactly-once delivery in most systems. You either have to risk missing a message, or having it delivered multiple times. If you're running idempotent code, you can always err on the side of redundant delivery without any ill effects.
My immediate thought was that some order processing step somewhere is not idempotent.
You should still try and make most things idempotent.
Let's say you're controlling a factory and you have a function that fills a reservoir. A naive way would be to define the semantics of the operation as "send enough liquid" and open the circuit for 10 minutes when activated. Since it takes 10 min to fill, all is good.
But what if your program crashes and you have to rerun. If the reservoir was already semi-filled, you will overflow it.
If the semantics of the operation is "send liquid until 'full' sensor indicates you're done", that's liberating. You no longer have to worry about overflowing the reservoir.
Also, don't let people guess what are those commands - create a menu that is run as soon as you clone the repo and shows all commands. I now use Invoke-Build [1] system for that and on all projects you type ib ? to get that list and later ib <task> to execute it (or any combination of it, i.e. ib DropDatabase, Run, Test). Basic tasks are named the same on all projects no matter the underlying technology.
Later you use the same command on CI server - its just another developer that runs ib Build, Test, Publish. Any more programming in CI yaml files then 1 to 3 lines per job is so wrong.
> 4. Delete code, delete tests, re-write stuff, re-write it again. Painful? Keep doing this till you get to the other side and feel liberation and empowerment, took me years to get past wincing at the idea of re-writing that thing AGAIN
With experience I've realized that rewrites are almost never a good idea[0]. They are time consuming and inevitably introduce new bugs (or reintroduce old ones), and the benefit is almost always marginal. If your architecture allows it, it's better to sandbox working legacy code and leave it as-is than constantly rewrite it.
My own policy is: if you're making significant changes to a unit of code anyway, clean it up or rewrite it; otherwise, make the smallest change necessary and leave it alone. If it ain't broke, don't fix it.
(Obligatory Joel Spolsky article on the topic that everyone has probably already read: https://www.joelonsoftware.com/2000/04/06/things-you-should-...)
[0]: ...with a possible exception if you're at a very large tech company with a lot of resources, where the testing/QA processes are a lot more thorough and it's possible to keep up with constant changes like this. But even then, seeing the amount of bugs introduced with each update to Facebook or Gsuite or any other large piece of tech, I'm skeptical.
What if your architecture is the problem? If you got the domain model wrong and you have deeply-nested complex types used throughout, you will never break free without a clean sheet rewrite. Certainly, you can refactor bits and pieces to conform to a correct model, but it's going to be an uphill battle the whole way. 10x if you are already in production with business state tied to the legacy schema(s).
Most of the horrible things I have seen at code review time have a root cause somewhere in poor domain modeling.
For example: Someone put support for 2 customer addresses as facts directly in the Customer type, so now you can't deal with the new edge case of 5+ addresses per customer, or model the idea that the address might be shared between more than 1 customer (and/or some other business types).
If you didn't model for 3NF/BCNF/DKNF up front, you might as well start over from the beginning in my experience. If your problem domain is not that complex, you can probably survive with something really shitty, but the moment you enter into 50+ types, 1000+ facts and 100+ relations, things are impossible to manage without strong discipline in this area.
I'd still prefer to do it incrementally if possible.
> if you're making significant changes to a unit of code anyway, clean it up or rewrite it; otherwise, make the smallest change necessary and leave it alone. If it ain't broke, don't fix it.
Don't be afraid to rewrite something when it needs it, and build knowing it's very possible you'll rewrite later.
If there are good unit tests, and it's a sufficiently small/decoupled piece of code, then rewriting is not so bad. These are all self-reinforcing things: small, decoupled code tends to be easy to rewrite; Testable code tends to be decoupled.
Systems always get more complex over time. The key thing is figuring out when your simple component is starting on the path to getting too complex, and taking the time to rewrite it as early as possible -- maybe it should be two smaller components, or maybe the entire organization of that are needs to be different. If you wait, and just keep adding "small" things, eventually you have a monster that's an order of magnitude more complex to deal with.
The other bit of this is writing code knowing you can (and may likely) rewrite it later. If you try to predict the future complexity early in design, 9 times out of 10 you will get it wrong, and you'll end up in a lose-lose situation: you have a overly-complex component to deal with, and it ends up needing a rewrite later anyway. Rewriting this is even harder because you have to undo the unnecessary complexity. This is also known as YAGNI ("You Aren't Gonna Need It").
The problem with this mentality is that everyone thinks they write “good unit tests”, but bugs are still found in production. :) You can’t use unit tests as justification for software being reliable; being battle tested in the real world is a much better indicator in practice.
I mostly agree with your other points, although I would still advocate doing these rewrites in as small of pieces as possible, in a way that’s as backwards compatible as possible, instead of all at once. I think you may be saying roughly the same thing, but this thread has shown that people have different definitions of “rewrite”, so it’s hard to tell. :)
My point on unit tests to support a rewrite is that it can catch a lot of edge cases, and you can remain reasonably certain you didn't break things if tests are still passing when you're done.
But otherwise I totally agree -- you can have as many tests as you want, with whatever coverage numbers you want, and it will still fall down in the real world. I don't focus on coverage or being dogmatic about TDD. I like to heavily unit test algorithms/regexes/etc -- anything with a defined input and output. At the same time, I hate testing 'glue' code (like an MVC controller), and would way prefer to rewrite it to be so simple it either works or it doesn't work at all (causing a big, obvious failure).
As bugs are found in the real world, in the ideal case write a test which basically guarantees that bug never happens again. Not always possible or takes such a massive effort that it isn't worth it, but usually it pays off.
> I would still advocate doing these rewrites in as small of pieces as possible, in a way that’s as backwards compatible as possible, instead of all at once.
Yes, exactly, that's what I was trying to get at. Big rewrites are hard to get approval/agreement to start, hard to actually do and often unsuccessful. Worse, a team with the mindset "ah, it's okay if we cut corners here, we're going to rewrite this whole thing someday" will rack up a serious amount of technical debt -- and make that big rewrite even harder. Ironically this makes a big rewrite even harder to start and succeed, but also more necessary.
Small continuous rewrites are a way of constantly improving quality, while avoiding a lot of these traps.
In some ways software engineering is so immature compared with other engineering disciplines.
I believe more in boy scout - "if you touch it leave it better than when you arrived".
:-)
What I've come to realize is that whenever people talk about wanting immutability, they really want value semantics. Value semantics can be achieved without immutability - the best example is Swift's structs. They can be both mutable or immutable, but they can never be shared, i.e. there can never be more than one reference to a single struct value. This means that modifying the value has no effect on other references, because there can be no other references.
This makes whether or not the value is immutable irrelevant, or at least not necessary. You get the same benefits - you can reason about your code locally and not worry about your changes modifying unknown parts of the program. Local reasoning is the end goal, and value semantics is the mechanism of getting to that goal. Immutability is only one way of getting value semantics.
Is that really worth it?
Also, creating the second vector should really look like "v2 = v.copy(x = x + 5); return v2.length". Or even just "return v.copy(x = x + 5).length".
This is the same model that Rust takes too. It doesn't eliminate mutability, only controls it. The argument that "performance doesn't matter anymore because computers are so fast" is a bad one. Efficiency is efficiency, and immutability will always be less efficient.
func newStructValue(s: Struct) -> Struct {
s.value = 5
return s
}
As a company grows the importance of each of these principles changes. As a project within an established company grows a similar maturation happens.
If you tried to adhere to all 'best practices' (principles or not) from day 1 of a project you'd be carrying a lot of weight that could crush otherwise good ideas.
Some principles, when presented without a timeline, seem contradictory and that can lead to fights on teams where developers are well meaning but less experienced.
I'm encouraged that this presentation includes dimensions along which you can slice the principles, but I encourage the authors to adopt a "stage" dimension as well that helps focus devs/learners on principles appropriate for 1. The stage they are in and 2. The immediate next stage.
Teams should come up with their own list of principles which reflect the team and adopt organizational ones that make sense. As organization ones will tend to be wider in scope and less prescriptive, it should be less of a problem to adopt them.
Basically teams should decide what's right for them based on their backgrounds and the organisation should provide general direction.
The principles become more useful when they are context sensitive, so principles lists are the next big feature (i.e one for your team, another for your organization)
This feedback is really useful and will inform the design of the app. If you have a good idea Ian of what you think is useful, please let me know how I can help. I think I've grokked the basics of what you're saying, but it would be good to lock it down.
That said, my mission is to make software engineering better for everyone. By capturing the best principles from the best in the world at what they do. And to organise and share that knowledge, freely.
It's currently missing features to organise principles well, but I'm working on it (discussion here: https://github.com/PrinciplesDotDev/principles/discussions/2...)
You can reach me through my profile or following me on @princples_dev (twitter) - I've only starting pushing this, but a lot more will be coming soon.
It's an open source project and it needs your principles to make this a reality so if you've got some to share, please do. Or if you have feedback, leave it here. I'm building it for you.
So you can download them from there and put it in an ereader or import it into an application that supports .md format.
In most teams, it's hard to point to those small behaviors because they have become habitual and you may have forgotten what they are. Forgotten the "why".
What I see in teams that don't work well, is teams don't have that guidance or don't share similar mental models to allow them to work effectively together. So you end up arguing at a higher level than the actual problem because you can't put your finger on what you believe.
Principles build capability. Take Redux, it has just three core principles: https://redux.js.org/understanding/thinking-in-redux/three-p... and from those core principles you can almost build the whole framework.
But more importantly it provides capability and confidence to people using the framework to extend if needed in a "redux" way. Instead of looking at the documentation, they know the authors intent. The "Why"
That said, the site is really in it's earlist phase and the principles will be improved over time with more in depth information as more people contribute. Eventually, a voting system will be in place so the best principles will come to the top.
For example, take "One single source of truth".
>Data should be held in one location, duplicates of that data should be by reference only.
>Why
>Changes to data are always propagated to the rest of the system.
>Mutations to the data need only happen in one place.
>Single source of truth means no data will be out of sync or fail to be updated.
>How
>Only allow data writes to happen in one location. Whether that be a call to a rest API, system call or other write actions.
>Don't allow data to be stored anywhere but the single source of truth.
That won't work in many situations. If you need to support high throughput OLTP use-cases but also want to support advanced filtering/sorting of large datasets, you may need something like a combination of a NoSQL DB with something like ElasticSearch. Eventual consistency is something you accept as a trade-off for enabling higher levels of performance.
https://principles.dev/p/one-single-source-of-truth/
> Exceptions > Highly distributed systems - Some systems rely on data consistency to be reached eventually or may never need to have accurate data.
I've written about exceptions here: https://principles.dev/documentation/#exceptions-optional
In general, exceptions should be quite broad and people can add these to the principles if they know of them.
They can also have higher priority "contradictory principles" which override the lower priority principles in certain cases.
Overlapping of principles create complex behavior so it is usually better to have a list of principles in priority order (this will be covered under emergent behavior in "principle-driven engineering") which can override the "single source of truth" principle.
An example might be: Engineers will know they should use single-source of truth, but as performance should be critical (or they have a specific business rule that states something must take less than 5ms) it will override that principle.
Does that answer your question or do you think it needs to be more refined than that?
I wonder if making the principles editable by the team would be a useful feature, so you'd be able to add your own exceptions to them for your particular use case.
Being able to stack rank principles is interesting. I think in practice, different principles apply to different use-cases and systems. There may be a certain set of principles for "critical" functionality pieces, but maybe a different set for features like reporting (such as staleness of data).
Amazon's leadership principles aren't specific to engineering, but they keep competing ones with tension between them all the same (e.g., "Bias for Action" vs "Insist on the Highest Standards").
Going to be a hard one to get right, so I'm taking my with that one.
I think there were suggestions about Time being an aspect of a principle. I wonder if this would be better represented by the SDLC stages, something like this might address the "time" part that allows it to be associated with type of work being done. Just a thought. I also wonder, at the same time, if this is too burdensome for a team or individual to do. Establishing your principles, IMHO, should be the most important thing you do, if you use them to guide your behavior.
I also have hope that when people establish what their individual principles (that define them) they then can ensure when they are job hunting or self reflecting on their current job that they what they are doing is aligned with who they are to ensure they will be satisfied in that role/job/company.
I just bookmarked the site. Excited to see this grow.
On the SDLC, it could work. The temporal aspect is something I need to think through in a way that's not too complicated. Getting feedback at this stage is beneficial, even if this hit HN a lot earlier than I was expecting.
Establishing your own principles is burdensome, but using others is not. Having access to everyone else's principles and being able to see what other successful teams use, makes it easy to take other people's capability and add it to your own. Imagine if you could see what principles Rob Pike or <insert favorite programmer uses> or the principles behind a library, framework or a particularly productive team? This gets me excited. It gives people the building blocks to make great things.
I totally agree that individual principles is extremely important. If not for the very fact that finding and being on aligned teams is an amazing experience for everyone involved. Happier, more productive teams.
I would love to have an informal chat with you, you get what I'm doing and it needs people like you to for this to succeed for the community. Drop me an email if you can take me up on the offer :)
Site checks out
I may take out the filters by technology choice for the time being, because there's nothing there.
I don't think anyone ever did.
I've a few principles for that: https://principles.dev/p/documentation-should-be-close-to-th... and “They Ain’t Gonna Read It” (not on the website, yet. But it is here: https://blog.nuclino.com/brown-m-ms-or-why-no-one-s-reading-...)
The trouble with the documents you've mentioned is they don't really create capability and it's really the social structure that enforces those values and principles as opposed to the documents.
With engineering it's different because it provides tangible value.
From a team perspective it can help you transfer mental models. Programming is an abstract activity that benefits greatly from those shared models. They build capability, help people learn rapidly, settle disagreements, bring the team together as one and are used in things like code reviews and filtering of technical decisions. People come back to them again and again - it's integrated. Then when a new member of the team comes a long, you're not going back to those discussions again and again.
As an individual. One of the reasons you look back at your principles to remind why you believe something or to be more convincing. They provide value, so they keep being used. It's also part of that persons identity. It defines what they care about and helps them join teams with people who are aligned.
So I agree to an extent - They Ain’t Gonna Read It... Unless it provides value.
I think principles are great, but like you yourself have said, they require context. I would especially be a valuable resource for junior and mid career engineers, of which I am one.
Not all of it would appear on the principles.dev, as I think the reflective nature would be best handled elsewhere. But acknowledging pros and cons on the website is very valuable.
The next big piece of work I have to do is on principle lists ( https://github.com/PrinciplesDotDev/principles/discussions/2...) and figuring out what features to include and where to draw the line is going to be tricky... I need to find the principles behind it, really.
It's interesting that you say they would be a valuable resource for a junior and mid-career engineers. I agree, it would. What I've found is it generally attracts people who are a) leaders (in some form or other) b) care about programming deeply.
You should iterate sequentially on a focused chunk of work at a time. Once that chunk has been completed THEN start on the next chunk of work. This is opposed to doing multiple chunks of different of work in parallel.
It's really a way of working as opposed to operating on code.
Someone already mentioned that "time" is absent here. Hence you cannot generalize. Each company should probably write their own principles and make it part of their identity.
There are no principles that make sense in every situation, there are no teams that will have the same principles.
Imagine being able to have a team and then adopt those principles easily into your team? That's the goal.
Why would you put in the work to warn people they'll have a bad time instead of just fixing the bad time.
I haven't put anything on the website intentionally, perhaps it's auto generated.
The only page that I haven't purposely optimised for mobile is the editor, as you can't really create principles on your mobile.
It would be hard to do it for every case, as principles interact together to create more complex behaviors. I'll be discussing this more in "emergent behaviors" in "principle-driven engineering" at some point. But the essence is architecture can arise from a few principles together. It's a bottom up approach to architecture where team members understand the "why" so there are shared mental models between the team.
slashdot(hn) effect at work ?