Famous Laws of Software Development (2017)
timsommer.be
timsommer.be
We have found that by changing our software/system architecture we have also inadvertently changed our organisation structure.
- Inverse Conway Law or just Roy's Law ;-)
Before we had four cross functional teams, working on a single application, everyone felt responsible for it, worked overtime to fix bugs etc, we had good communication between the teams.
But after we switched to microservices the teams became responsible for just a part of the system, their microservice(s). Whenever we had an outage, one team was left to fix it, the others just went home. They stopped talking to each other because they didn't share any code, no issues... they stopped having lunch together, some things got way worse in the organisation, all sparked by a 'simple' architectural change, moving to microservices.
I ended up leaving during the peak of all of this, but in an exit-interview, a director asked me about the problems this was causing.
Was that accompanied by any growth in company size? I've found that this happens when a group grows past about 15 people even if the structure doesn't change.
Sounds like the organization needs to do other things to keep people from getting siloed, though that gets increasingly difficult at scale. Well-defined SLAs (along with monitoring and reporting of those SLAs) are also necessary so that microservice failures can be understood in the right context.
Honestly, this sounds like an improvement.
It is perfectly fine that the people communicate, and even helping each other to improve tech skill should be encouraged; however, decisions about their respective products should be contained within each team, with clearly defined interfaces and usage documentation.
In order to make those decisions and define the interfaces, you need to know a lot about how your software is going to be used. That will be much easier if you have good communication with the other teams and understand their goals and motivations.
The main advantage of distributed system, and particularly microservices, is the ability to have each system completely independent: individual components can be written in different languages, running on different platforms and use completely independent internal components. Basically, it is just like using an external library, component or service: the authors provide documentation and interfaces, and you should be able to expect it to behave as advertised.
If you just implement all requests directly, you're for sure going to end up with a horrible interface. You should approach API design the same way that UX/PM approaches UI and feature design: take the time to understand _why_ your partner teams/engineers are requesting certain changes and figure out the right interface to address their problems.
Edit: I just noticed that "PM" in parent comment. Basically, product managers are not just for UI and customer-facing products.
But yes, you can never just do what "the users" ask for. The best way to understand what they need is to be in conversation with them. Silo'ing everyone up isn't going to make the overall product -- which has to be composed of all these sub-components working well with each other -- any better.
Oh it absolutely can; it's just that it usually is not a good idea. But I'm not talking about the process of reaching that decision, I'm talking about the responsibility to reach them. Functional and technical decisions are separate, but in most cases should be defined in conjunction.
> Silo'ing everyone up isn't going to make the overall product
This is true for certain types of product, and less so for others; you need to clearly understand what type of product you're building, and be ready to adapt as it changes (or you have a better understanding of it). In a nutshell, the more compartmentalised a product is, the isolation between the teams becomes more beneficial. Which brings us full circle back to Conway's Law.
Of course you're talking about the process. Your claims that "direct communication between teams is not the right method for that," and "communication between teams working on discreet, separate units can become detrimental," for instance, are about process.
I don't think this conversation is going anywhere, but from my experience, lack of communication between teams working on discrete, separate units (that are expected to be composed to provide business value), can become detrimental. And that's about process.
But culturally, it would be nice if people helped each other out from time to time...
Or perhaps your statement is correct, but in the real world components are never sufficiently separate. So, while your statement may be correct by definition, it is not useful.
I can imagine there being a normal distribution of 'separateness' of software and the rare top tail-end of the distribution gets it perfectly right, most are in the middle somewhere between 'service oriented architecture' and 'ball of mud' and some are just at plain ball of mud.
> in the real world components are never sufficiently separate
But the separation of components is not an issue of "real world", it is a function of design and implementation. It is absolutely to the developers how independently the components will be implemented; if there is no way to test them in isolation, then they are not really separate components.
Take this website, and your browser, as an example. They are obviously connected, as you're using the latter to access the former, but they are completely independent: you can access other pages with the browser, and you can use other methods (other browsers, or curl, or Postman etc) to access this page. Each can be tested separately, and even when they are used in conjunction they don't directly depend on each other, but rather on a set of standards (HTTP, HTML, CSS etc) understood by both.
This actually sounds very dysfunctional, but with the type of positive PR spin that product / executive management wants, basically anyone who believes “cross-functional” is anything more than a buzzword.
Would love to know what the engineers thought about working in that environment (which sounds like a monolithic, too-many-cooks, zero specialization situation likely negatively affecting career growth & routine skill building).
This part is often much worse than the overtime part, because it means you’re expected to sublimate your personal career goals in favor of whatever arbitrary thing needs done for the sake of the cross-functional unit.
When I hear someone describe communication between cross-functional team members as “good” or “effective,” then I know it’s a big lie, and most probably it’s a disaster of overcommunication where product managers or non-tech leadership have a stranglehold on decision making when really engineering should be autonomous in these cases exactly according to independent specialization.
"With a sufficient number of users of an API, it does not matter what you promise in the contract: all observable behaviors of your system will be depended on by somebody."
I.e. any internal implementation details that leak as behavior of the API become part of the API. Cf. Microsoft's famous "bug for bug" API compatibility through versions of Windows.
Http://www.hyrumslaw.com
That sounds nice in theory, but doesn't really work in practice. If you're building infra and a core piece of your company's product relies on these undocumented behaviors, you can't just change the behavior and shrug your shoulders when the whole product breaks. Similar if you're providing an external API to users/customers, you can't just break their stuff without worrying about it.
Experienced recently as a consumer of an API when letsencrypt made a breaking change to implement the protocol correctly. Broke my code which relied on their original incorrect implementation.
I can testify to this personally, having worked at a payments processor and accidentally broken integrations. The business, as it should have, had little tolerance for me changing a depended upon API, even though it was not documented
[1] http://www.globalnerdy.com/2007/07/18/laws-of-software-devel...
[2] https://exceptionnotfound.net/fundamental-laws-of-software-d... (not so good but with solid discussions from HN https://news.ycombinator.com/item?id=11574715 )
And more:
[3] https://embeddedartistry.com/blog/2018/8/13/timeless-laws-so... (a whole book titled Timeless Laws of Software Development)
[4] https://www.red-gate.com/simple-talk/opinion/opinion-pieces/...
[5] https://www.netobjectives.com/blogs/some-laws-software-devel...
Organizations that are too isolated will tend to create monoliths. Organizations that are too connected and too flat will tend to create sprawling spaghetti systems. These two cases are not mutually exclusive. You can have sprawling spaghetti monoliths. This is also one of the dangers to having one team work on several microservices; those microservices will tend to intermingle in inappropriately complex ways. Boundaries are critical to system health, and boundaries can be tuned by organizing people. Don't worry about Conway's law, leverage it.
The first 90% of the code takes the first 90% of the time. The remaining 10% takes the other 90% of the time.Another kind of corollary: "If the business will go under if we don't get this done by X, then we probably need a new business plan, not faster development".
These are rules of thumb and there are definite places where they don't hold, but I've found it genuinely useful to consider when the inevitable tough questions start to get asked.
The iteration is stopped when the software has enough features and an acceptable level of bugs to be considered complete. What complete is depends entirely on the field of the software. For a proof of concept software we can stop after the first iteration, but for a safety critical software we might need 3, 4, or even more itarions.
Unfortunately, lots of us bid against people who over-promise. By the time the project is obviously behind schedule, it's too late, and the client can't switch to someone else.
[0] I think it is fun. Your milage may vary.
Interestingly, Price's law seems to indicate 10x developers exist because if you have 100 employees, then 10 of them do half of all the work.
This idea is particularly critical when it comes to things like layoffs. If they get scared and leave or they are let go for whatever reason, the business disproportionately suffers. Some economists believe that this brain drain has been a primary cause in the death spiral of some large companies.
Or 0.1x developers exist...
To take things further, make a bell curve chart. Put 50% of the area under the top 10%. Now, divide up the rest however you'd like. The only way to make this happen is for a huge percent to not only contribute zero, but to be actively hurting development to an extreme degree.
I have never found a 100 person company where 60% or more of the company was contributing absolutely nothing. I have never seen a company where a large number of people were actively harming the company and the company survived.
"To survive" is a temporal measure. It's pretty common for companies to survive on a successful product (or group of products). The rest of the company was a shell and revenue sink for that line.
- the distribution of productivity of devs in an organization of N * N devs can be approximated as: N devs who are "Nx", and the rest of the devs are "1x" (Price's Law, assuming a binary distribution for simplicity)
- the value of "x" is constant for all sizes of organization (if it were relative "some are 0.1x" would be a change of units, not an abandonment of reality)
This would yield the extremely surprising result that the total dev production of an organization scales quadratically with the number of devs, so what am I misunderstanding?
Rather than quadratic scaling, we're dealing with scaling by root. This actually meshes very well with the "mythical man month"
If we almost double from 100 devs to 196 devs, we only go from 10 to 14 devs doing half the work.
We've already accepted that 10 devs were doing half the work of 100 devs. We've also accepted that those devs must be giving it their all. So, doubling the devs, but only getting 4 new people to fill the doubled top 50%. Either we have some new 20x devs or the actual amount of work hasn't increased at the same rate.
I would still say that is probably incorrect though. The "mythic man month" doesn't apply to total work done -- only to total useful work done. As the social complexities increase, the ratio of other work decreases, but the those top developers will still have to carry both increases (to at least some degree) in order to still be doing half the work.
I suspect that as the social overhead increases, you should see three interesting cases. Those who can deal with the social overhead more quickly, so they have more real work time to compensate for being slower at it (potentially bumping a 5x dev with better social strategies higher). You could see the opposite where a 10x dev simply loses all their time in meetings. You could also see where a 1x dev with better social strategies handles most of a 10x devs social workload so that dev can instead focus on coding (it's rare, but I've worked on teams with 1-2 devs who did little except keep the team productive by fending off the bureaucracy).
Sorta buried the lede there, eh?
If someone has found the Russell's Teapot of companies that strays so far into absurdity while still being true, then let them bring forth the proof.
The reason it's a really great idea is that it says you should engineer in favor of resilience, which is an important form of robustness. And at the same time, "strict in what you send" means "don't cause trouble for others.
However there are cases where "fail early" is more likely to be the right thing. Here are a few:
1 - Backward compatibility can bite you in the leg. For example, USB Type C (which I love!) can support very high transfer rates but when it can't it will silently fall back. So you could have a 40 Gbps drive connected to a 40 Gbps port on a computer via a cable that only supports USB 2 speeds. It will "work" but maybe not as intended. Is this good, or should it have failed to work (or alerted the user to make a choice) so that the user can go find a better cable?
2 - DWIM is inherently unstable. For users that might not be bad (they can see the result and retry) or terrible ("crap, I didn't mean to destroy the whole filesystem").
I see these problems all the time in our own code base where someone generates some invalidly-formatted traffic which is interpreted one way by their code and a different way by someone else's. Our system is written in at least four languages. We'd be better off being more strict, but some of the languages (Python, Javascript) are liberal in both what they accept and generate.
This aphorism/law was written for the network back when we wrote all the protocol handlers by hand. Now we have so many structured tools and layered protocols this is much less necessary.
Every program attempts to expand until it can read mail. Those programs which cannot so expand are replaced by ones which can.
Any social media company will expand until it behaves like a bank; receiving deposits and making loans to customers (not necessarily users).
(HN discussion: https://news.ycombinator.com/item?id=19216077 )
Some people, when confronted with a problem,
think “I know, I'll use regular expressions.”
Now they have two problems.
Some programmers, when confronted with a problem, think
"I know, I'll solve it with threads!"
have Now problems. two they 2. Exactly-once delivery
1. Guaranteed order of messages
2. Exactly-once delivery
> There are only two problems in computer science, naming things, cache invalidation and off by one errors.
I've got to say modern languages with foreach() have been amazing (makes me feel old when I consider a 20 year old widely used language 'modern').
"Any application that can be written in JavaScript, will eventually be written in JavaScript."
Any sufficiently complicated C or Fortran program contains an ad-hoc,
informally-specified, bug-ridden, slow implementation of half of CommonLisp.I’ve been looking at some disused AI systems, which were all written in Lisp back in the day.
In an attempt to remain relevant, at one point in the early 2000s someone tried porting one of them to Java. By first writing a Lisp interpreter in early 2000s Java. So the system had all the old dynamic Lisp programs as giant strings, embedded in a fugly static class hierarchy.
Any sufficiently complicated concurrent program in another language contains an ad hoc informally-specified bug-ridden slow implementation of half of Erlang.
If this sort of content is the sort that this community increasingly selects for then it is perhaps time to look for fresh pastures. ( I don't however know if this indicative of HN's current community or just an 'accident' - I'm sure there have always been examples of poor quality near the top at times).
There is a lot of back and forth in the comments about software design and workflow practices. I think this kind of discourse is extremely valuable.
It's hard to find the original piece of art, but my uncle had this hanging in his office for a long time, and now it's hanging in mine.
I transcribed it in a gist so I had access to them for copy/paste.
https://gist.github.com/sorahn/905f67acf00d6f2aa69e74a39de65...
(Those pictures were from an ebay auction before I got the actual piece)
... then it grows even faster.
https://tools.ietf.org/html/draft-thomson-postel-was-wrong-0...
(Don't know if it has a name)
A related rule: Design software for maintenance, not initial roll-out, because maintenance is where most of the cost will likely be.
An exception may be a start-up where being first to market is of utmost importance.
Other rules:
Don't repeat yourself: factor out redundancy. However, redundancy is usually better than the wrong abstraction, which often happens because the future is harder to predict than most realize.
And Yagni: You Ain't Gonna Need It: Don't add features you don't yet need. However, make the design with an eye on likely needs. For example, if there's an 80% probability of a need for Feature X, make your code "friendly" to X if it's not much change versus no preparation. Maybe there's a more succinct way to say this.
From what I know, the "*s'" thing works mostly for plural nouns. For singular, it only applies to classical & religious names ending with "s" ("Jesus'", "Archimedes'" etc).
I am not an English native so I may be completely off. Feel free to rage :)
Both are acceptable, apparently.
Here's an example of the lack of consensus:
Either is acceptable: https://data.grammarbook.com/blog/apostrophes/apostrophes-wi... https://owl.purdue.edu/owl/general_writing/punctuation/apost...
Chicago vs AP style: https://apvschicago.com/2011/06/apostrophe-s-vs-apostrophe-f...
APA style suggests appending the extra 's': https://blog.apastyle.org/apastyle/2013/06/forming-possessiv...
Thanks for the links. Plenty of educational value there!
Because that would be an improvement, or not much of a change?
However that didn't stop the Beatles from using "In an Octopus's Garden" as a song title. (Note that the song is about a single Octopus). I would suggest that it depends on whether you intend to explicitly repeat the 's' when speaking.
Plurals of words ending with an 's' are an occasional minefield. You sometimes hear people smugly insist that the plural of Octopus should be Octopi, only to have someone even more smugly point out that Octopus is from Greek, not Latin, and so it should be Octopodes. Meanwhile the rest of us just continue to use Octopuses....
* One river's fish.
* Jesus's fish.
* Many rivers exist.
* Many rivers' fish.
The name "Brooks" unfortunately fits both the second and fourth of these examples, making it even weirder.
> Redundancy is bad, but dependencies are worse.
https://yosefk.com/blog/redundancy-vs-dependencies-which-is-...
> Always code as if the guy who ends up maintaining your code will be a violent psychopath who knows where you live.
https://stackoverflow.com/questions/876089/who-wrote-this-pr...
(I first heard that for XML, and since have heard it for others. Was very funny for XML though. I also know its not really a law.)
A programmer can accurately estimate the schedule for only the repeated and the redundant. Yet,
A programmer's job is to automate the repeated and the redundant. Thus,
A programmer delivering to an estimated or predictable schedule is...
Not doing their job (or is redundant).
"Everything breaks, all the time" - Dr. Werner Vogels CTO Amazon.com
> Given enough eyeballs, all bugs are shallow.
Just count of viewers doesn't help. The owners of these eyeballs need both motivation to look for these bugs, and expertise to find them.
> The power of computers per unit cost doubles every 24 month.
Slowed down years ago.
> Software gets slower faster than hardware gets faster.
It doesn't. If you'll benchmark new software on new PCs versus old software on old PCs processing same amount of data, you'll find out the new one is faster by orders of magnitude.
Input to screen latency might be 1-2 frames slower, because USB, GPU, HDMI, LCD indeed have more latency compared to COM ports, VGA, and CRT. But throughput is way better.
> Premature optimization is the root of all evil.
Was probably true while the power of computers doubled every 24 month. It doesn't any more.
For example, if you have a service with near 100% uptime, any other service which relies on it may not be able to handle errors or unavailability. Introducing errors in a controlled way can help make the dependencies more reliable.
As another example, being liberal about what you accept can sometimes result in security flaws, since different systems might be interpreting a message differently. Rejecting malformed input can be a beautiful thing.
If you control all the clients and servers using a protocol, it does not apply to you. You're better being as strict as possible.
More concrete example: suppose that an incoming HTTP request contains CRLFLF. To the proxy, “liberal in what you accept” might mean that this is interpreted as CRLF CRLF, which is the end of the request header. To the authoritative server, perhaps the second LF is silently discarded and ignored. Voilà: instant cache poisoning attack.
Any sufficiently complicated C or Fortran program contains an ad-hoc, informally-specified, bug-ridden, slow implementation of half of Common Lisp.
And Beneford's law of controversy [2], which I see around monorepo vs polyrepo, language choices, tabs vs spaces, etc:
Passion is inversely proportional to the amount of real information available.
[1] https://en.wikipedia.org/wiki/Greenspun%27s_tenth_rule [2] https://en.wikipedia.org/wiki/Gregory_Benford
> Programmers waste enormous amounts of time thinking about, or worrying about, the speed of noncritical parts of their programs, and these attempts at efficiency actually have a strong negative impact when debugging and maintenance are considered. We should forget about small efficiencies, say about 97% of the time: premature optimization is the root of all evil. Yet we should not pass up our opportunities in that critical 3%.
Premature optimization isn't bad, premature micro optimization is bad. You should also be thinking about optimization that results in better architecture and architectural decisions that make it easier to optimize in the future.
The full quote makes me think: "You should identify the critical paths of your system early." The shortened quote makes me think: "Deal with performance later."
Pretty big difference in meaning.
Most decisions are small, but can lead to compounding effects. Personally, I think one should also avoid premature pessimissation as well. No one in their right mind would use bubble sort over quick sort, for instance (not saying quick sort is the best algorithm, but it's better than bubble sort). One pet peeve I have in C++ is when I see people initializing a std::string with a literal empty string instead of using the default constructor. The default constructor is usually a memset or initializing 3 pointers. Initializing with an empty literal involves a call to strlen, malloc and strcpy. I've yet to see a compiler optimize this. May not seem like a big deal, but considering one of the most frequently used data types is a string, it adds up a lot. Most of the applications I've worked on show std::string methods and constructors as hotspots when profiled (back office financial systems).
I agree one should avoid premature micro-optimization, but that you can also avoid premature pessimissation.
Some choice tweets:
Cloud is not ready for enterprise if is not integrate with single server running Active Directory.
Fact of useless: 90% of data is not exist 2 year ago. Fact of use: 90% of data is useless.
In devops we have best minds of generation are deal with flaky VPN client.
For increase revenue we are introduce paywall between dev team and ops team.
> It is much better, and more and more implemented as such, to deploy teams around a bounded context. Architectures such as microservices structure their teams around service boundaries rather than siloed technical architecture partitions.
> So, structure teams to look like your target architecture, and it will be easier to achieve it. That's how you defend against Conways's law.
- noone reads open source code
- those who read do not understand it
- those who understand don't file bug reports.
- those who file bug reports file them for their own issues coming from misunderstanding/misapplication of the software, not actual bugs.
* All software has bugs, no software is inefficient
* A programmer's work is never done
Also, I think Murphy’s law should be removed, it’s less true than the other laws here.
I read a fantastic article many years ago in the Atlantic where the author was analyzing and deconstructing an airplane crash, and in it was a paragraph about how Murphy’s law is completely backwards, and in reality if things can go right, then they will. Things will almost always go right unless there’s no possible way they can, in other words only the extremely rare alignment of multiple mistakes causes catastrophes. Can’t remember if the author had a name for the alternative Murphy’s law, but I believe it, especially in software. We get away with crappy software and bugs & mistakes all over the place.
We can extrapolate from "Anything bad that can, happen will happen", and get the statement: "If something can physically happen, given enough time, it will eventually happen."
I like to think its sort of a very tangential sister idea of the mediocrity theory.
Here's the article I was thinking of. Totally worth the read, aside from discussion of Murphy's Law...
https://www.theatlantic.com/magazine/archive/1998/03/the-les...
"Keep in mind that it is also competitive, and that if one of its purposes is to make money, the other is to move the public through thin air cheaply and at high speed. Safety is never first, and it never will be, but for obvious reasons it is a necessary part of the venture. Risk is a part too, but on the everyday level of practical compromises and small decisions—the building blocks of this ambitious enterprise—the view of risk is usually obscured. The people involved do not consciously trade safety for money or convenience, but they inevitably make a lot of bad little choices. They get away with those choices because, as Perrow says, Murphy's Law is wrong—what can go wrong usually goes right. But then one day a few of the bad little choices come together, and circumstances take an airplane down. Who, then, is really to blame?"
Of course, regardless of which way you interpret Murphy's law, the law itself and this alternative are both hyperbolic exaggerations. The main question is more of which way of looking at it is more useful.
In terms of thinking about safety, it seems like both points of view have something important to say about why paying attention to unlikely events is critical.
Murphy's law is a favorite of mine because it's the perfect driving board for conversations about infinite probabilities and aliens and simulation stuff.
I feel like Murphy's law as stated captures that idea adequately. And it's certainly true if the event probability really is non-zero. Sometimes, though, we can calculate event probabilities that are apparently non-zero based on known information, but are zero in reality.
One example in my head is quantum tunneling. Maybe this is along the lines you're talking about? And this is the way my physics TA described it many years ago, but caveat I'm not a physicist and I suspect there are some problems with this analogy. He said you can calculate the probability of an atom spontaneously appearing on the other side of a solid wall, and you can calculate the same (less likely) probability of two atoms going together, therefore there is a non-zero probability that a human can teleport whole through the wall. The odds are too small to expect to ever see it, but on the other hand, with the amount of matter in the universe we should expect to see small scale examples somewhat often, and we don't. There may be unknown reasons that the probability of an event is zero.
Murphy himself was unhappy abut the common interpretation of his law, which is negative rather than cautionary, implying a vindictiveness to exist in inanimate objects and the laws of chance.
> but more about anticipating the failure, and designing your code/product/system for the worst case scenario
Which was his intent. IIRC the phrase was coined while working on rocket sleds for a land speed record attempt. He was essentially trying to achieve "no matter what happens we want to maximise the chance of the pilot being alive afterwards, if some of the equipment survives too that is even better" and promoting a defensive always-fail-as-safely-as-possible engineering stance.
Are you sure about that? Murphy's actual statement was negative and not cautionary. He was criticizing a person, not saying something cautionary about the nature of the universe.
https://en.wikipedia.org/wiki/Murphy%27s_law#Association_wit...
"Any technology that surpasses 50% penetration will never double again (in any number of months)."
http://www.norvig.com/norvigs-law.html
But most certainly that is not how the law was meant to be interpreted.
* Putt's Law: "Technology is dominated by two types of people, those who understand what they do not manage and those who manage what they do not understand."
* Putt's Corollary: "Every technical hierarchy, in time, develops a competence inversion." with incompetence being "flushed out of the lower levels" of a technocratic hierarchy, ensuring that technically competent people remain directly in charge of the actual technology while those without technical competence move into management.
In the Peter model, everyone gets (or tends to get) promoted until they reach a job they can't do, and they stay there. Thus everyone will (tend to) be incompetent. In Putt's model, the technically incompetent get promoted, and those at lower levels are competent.
Putt's does sound more like the way the world works...maybe. Peter's has always sounding convincing to me, yet the world evidently isn't so bad as that.