Tips for beginning systems and software engineers
ilya-sher.org
ilya-sher.org
I'm no TDD or automation advocate - however I do see that this tool is essential to modern software development, and the only way to learn to use it, is by practice.
I would encourage every engineer who's covered the basics to go full-in on TDD with 100% test coverage for a few days or weeks. Not because it will make you more productive or less bugs (you will be less productive), but it will change your thinking of writing code (similar to how learning Lisp will have a massive impact on your thinking of structuring code). TDD + 100% coverage is an extreme, which is worth experiencing to take learnings from it.
Later down the road, when you've gotten into the habit of developing with no, or barely any automation, it will be much more difficult to learn to fit this tool into your coding routine.
So I think it's great to mostly ignore that, since it's contentious, and focus on that core question: how do I know that the system works correctly?
This also makes it obvious why type systems are a glorious ally, in addition to test suites and proofs.
And it's a core question because if I don't have grounded confidence in the system's correctness, then that's likely to be anxiety-inducing, irresponsible, and sometimes dangerous.
If we are talking about the later stages of the life of a software project then I completely agree with you.
But if you just started a new project and you are still in a constant state of change then a large test suite will slow you down because you have to adapt all your tests all the time. In the worst case you might even stop to improve your software just because updating your tests is too much work.
Moreover, creating that test suite after the project has grown is likely to be tedious, too... and often requiring serious refactoring to accommodate testing practices.
The XP folks were always all about changing requirements and flexible code, and unless they're all deluded, it's not true that testing necessarily hampers a new project.
I have been in a new project where progress was slowed by all the tests. But the issue I have with "I will write tests later, after things have stabilized" is that, in my experience, that day never comes.
There is always a feature that would add value, or some fire to be put out.
Of course, you could arbitrarily pick a date or time to start testing, but as some one who was responsible for a smallish code base (30k lines, maybe 2-3 man years) with no tests, I can tell you that it is hsrd to start that. You have to either start testing when building a new piece of software (which is easy, but you really want to have assurances about the working of your existing code), or try to apply tests to previously untested code, in which case you are unraveling quite a bit of complexity and possible untestable/difficult to test constructs.
I don't understand what the rush is all the time. Sit down with a pad/pen or whiteboard and think about the design. the last mobile project i started, I sat down and wireframed on paper. Tests came naturally and didn't fight the design.
Then, tests are simply your hypotheses.
I would place that kind of programming in a different class. Small, "one-shot" problems probably don't need a lot of planning.
That's the experimental / exploratory class of problems where you're not interested in the correctness of the process but results. AKA risk reduction.
For the other classes where there is "The Process" of validation, testing and approvals, a little planning up front saves some churn on the back end.
I have the same concern about iterative development. Seems like a lot of teams jump right into a coding sprint/iteration/cycle w/o a plan.
That's exactly when you need automated test suites, if your pushing multiple versions a day, then automated tests are really required.
They can be read almost like project requirements, so they double as validation for solving the problems in the reqs.
They only use the outermost APIs of your code, which will change much less than internal module APIs, and so won't inhibit changes and experimentation as much.
They enable you to automatically test your code in the "real" scenarios that your project reqs describe, instead of just module-level tests that may not catch bugs in API usage of those "real" scenarios.
Later on in the project, as larger internal modules crystallize, you isolate those modules with their own outer APIs and write automated tests for their APIs.
Yes, this has downsides. High level white box testing exclusively during early stages obviously won't catch all your internal error cases early. However, I have found them to have superb ROI, especially in projects where time constraints have required pragmatic approaches.
YMMV.
Then again, I skew towards integration rather than unit testing. So I am not locked into an internal design, but I know outward facing behavior is as I expect it to be.
Possible solutions to this are TDD, or trying to write your tests in such a way that they would fit any implementation of your interface. For interface design, outside-in TDD is an interesting tool.
In any way, your problems with tests at early stages probably come from lack of practice. Doing less tests is certainly not going to help.
I hope its omission is because OP thinks it goes without saying. It doesn't though, just the other day I read on HN that people have colleagues who don't use any source control.
Don't agree, imo a good automation service is abstract enough that it can be hooked into your cycle or at least appended to it. Automated tests and CI is a prime example.
I'm ready for the down votes, but the most important aspect of TDD is job interviews (or blog posts), telling others you do it even at home on your pet project is a big plus somehow. My guess it is the best way to communicate you care about quality, even if equating the two is obviously a half truth. Again Norvig vs. Jeffries tells me practices go a shorter way than actual practicing and ability.
Automated test systems, in my experience, have sucked pretty hard. Often written by Q/A engineers frustrated at not being "real" engineers [yeah I know, toxic culture], and just not written well. Like the test systems I could never install on my workstation or even get running on my own, in order to reproduce problems and debug them. Or the systems that would assign "Priority 0 wake-up-in-the-middle-of-the-night" issues to failures in the test framework itself. Test frameworks are best kept as simple as possible; be wary of architecture astronautics.
I like the "TDD is nice, use it when it's useful" attitude. TDD as a tyranny works about as well as any other silver bullet we've seen in software development over the years (that is, not very well).
Anyway maybe something like magic bean or some other "looks useful but turned out not to be" metaphor would be better, not sure which one though.
[Depending on your mythology, you can kill a werewolf with a silver sword, and probably other silver objects. Poul Anderson has an interesting treatment of werewolf vulnerabilities in his collection _Operation Chaos_]
"if you've just started out and ... you don't see a single person who's smarter than you"
Many people starting out think they're much 'smarter' than they really are - experience often counts for much more.
Of course, you could also have landed in a real dead-end company.
I couldn't get them to guide me when I needed it, instead, my PR's were rejected saying: "Don't see any value", or "This feature is incomplete and undocumented". Also, when I did specify in my comments "Feedback appreciated", I felt that my work was nitpicked way more than of their own, which I saw as double standards. In the end, my self-esteem went down a lot, I couldn't keep working with them and left. I got back to my previous contract work where I experiment and continue working on the same project (PHP), working on an old legacy system that needs improvements. How do I keep learning and continue work as a sole developer? One way is to learn from books, but I realize learning from smarter people has tremendous value.
At the same time I feel that maintaining a live legacy system can contribute to my experience improving the existing projects and provide value for the company (bugfixes, refactoring and improving stability).
I'd appreciate an opinion on that.
I advise you to start collaborating to Open Source projects, don't be shy. Everyone is welcome and I've personally found OSS devs to be a friendly bunch(exceptions always apply though). That will be very valuable feedback from people of different levels of experience and people in different positions. Leverage this feedback to improve yourself, but be ready to flush your pride down the drain.
Feedback can also come from "oh, that doesn't work as well as I'd expected".
Feedback from other people is usually faster and more efficient (so I agree with the recommendation), but isn't the only game in town.
If I come back to my own code a couple of months later, and it isn't immediately obvious how its working, then its a good time to refactor it and think about what you would have expected to make it more obvious. My code gets better as a result.
(I understand these are suggestions and we are in the realm of personal opinions).
Even as a Systems/Embedded programmer it's hard for me not to consider HTML/Javascript a must-have, this day and age (not a nice-to-have).
Even in embedded/industrial systems nowadays it's perfectly normal to have a maintenance interface based on a lightweight web server.
A better mix would be:
Python, Javascript, SQL, and Regular Expressions
This mix would cover integration, ux, persistence, and madness.
I dont know any of these. I know they exist, i know of the concepts, but no specifics. I would argue that these are good to know, and a must if you are working in a domain where this matters. For design, architecture and implementation in a high level language when the problem do not touch these areas this is not a MUST have.
1 and 2 you won't need every day. You can probably even make a career out of software not knowing them. But every software related domain will have times where knowing them will help you, and where not knowing them risks an inefficient and slow implementation and no way for you, individually, to fix it in any sort of reasonable time; you'll either have to learn it then, or you'll be reliant on someone who does know it fixing it for you.
I've had to address performance issues where knowledge of common data structures and algorithms' time and space complexity (and/or being able to look at an algorithm and realize "...this is O(n^2). It only needs to be O(n). WTF", and fixing it) was the key in Javascript, Erlang, Python, and Java. Exactly what 'high level language' do you feel you won't eventually need this stuff in?
1) You don't strictly need to know Big O notation in order to not introduce performance bottlenecks, or mitigate them when they arise; but you should be able to use common sense resource wise, when developing things in any problem area. In other words: you should conceptually understand how computer systems work (which often implies you'll sooner or later run into Big O notation). Knowing Big O notation can also help you communicate with your peers about problems, if they know it too. 2) For a large number of problem areas, this is knowledge that is near useless I think. Know that they exist, and that they might ever help you. But for your average problem domain, you don't really need concrete knowledge there. It bothers me how much emphasis is put on this in so many job interviews, while it's hardly ever relevant in many of those cases. 3) I'm not sure how that knowledge will really help you. 4) I think this is actually the most important thing of all four. Knowledge of the kernel's main system calls will help you tremendously when your process (or any process in your stack) "just hangs" sometimes and you need to resort to systems such as strace, dtrace, sysdig and what not. That time will come, sooner or later.
I was surprised to see this so far down. I, on a somewhat regular basis, go back to system calls. Usually it's related to needing to strace a process, but it is also helpful when thinking about what's going on when I'm writing some high-level code and how that will be translated into calls to the kernel versus user-space execution.
Oh, but I really like watching the algorithm sorting gifs. They're neat.
3 is mostly irrelevant unless you're doing embedded devices or fast maths. If you're writing Java it might be more useful to know JVM bytecode opcodes and underlying CPU opcodes are completely irrelvant to you.
4 I think you need to understand the existence of the system call mechanism; then learning the API you realise how much of it is just syscalls (e.g. open/read/write/close).
I can write Windows code quite happily without understanding its syscall mechanism.
Big O's not too hard. I would imagine that most programmers have an intuitive feeling for them. For example:
If I need to use a loop, that's going to slow things down (formally, going from O(1) to O(n)).
If I keep putting loops inside loops, it's going to get really slow (formally, going from O(n^x) to O(n^(x+1))).
Each element requires all the previous to be processed again. That's not going to work (formally, reaching O(2^n)).
Maybe some fancy datastructure would speed this up (formally, going from O(n) to O(log n)).
> 2) common algorithms and their time and space complexity
I think this just boils down to appreciating what kind of work must be going on in some algorithm. For example, sorting a list is surely going to loop through it (i.e. O(n)); actually it's O(n * log n), but that's pretty close.
I would imagine that the most effective knowledge for a complete beginner would be bad usage patterns for a bunch of common datastructures. For example, a loop which appends to the end of a linked list is a bad idea, since finding the end of the list requires looping through all the nodes. This is a loop inside a loop (AKA O(n^2)) when we intuitively know that only a single loop is required (AKA O(n)).
That would be indicative of a bad/naive linked-list implementation, wouldn't it? It's not that much overhead to maintain a wrapper with front and back pointers.
There are some widespread cases of these naive datastructures which don't fit that model though. For example, Lisp makes heavy use of "cons" to pair up two values. By nesting these pairs, we can get arbitrary tree structures, and this is how lists are implemented, e.g. "(list 1 2 3 4)" is equivalent to "(cons 1 (cons 2 (cons 3 (cons 4 nil))))". These structures are heavily used in Lisp (since it does so much "LISt Processing"), and since the language is so dynamic it's hard to compile it away.
Haskell's default list type works in this way too. There, the problem is compounded by immutability: changing the start of a list is easy, since we can re-use the pointer (immutability allows a lot of sharing). For example, if "list1" is "[1, 2, 3]" and "list2" is [9, 2, 3]", they can share the "[2, 3]" part:
1
^
|
list1: cons * *----+ 2 3
| ^ ^
V | |
cons * *--->cons * *--->nil
^
|
list2: cons * *----+
|
V
9
5
^
|
list3: cons * *--->cons * *--->cons * *----+
| | |
V | |
1 | |
^ | |
| V |
list1: cons * *----+ 2 3 |
| ^ ^ |
V | | V
cons * *--->cons * *--->nil
^
|
list2: cons * *----+
|
V
9
class List<T> {
ListNode<T> head;
ListNode<T> end;
int size;
}
class ListNode<T> {
T data;
ListNode<T> next;
// possibly, if it is doubly-linked
ListNode<T> prev;
}Bad repetition is when you or other engineers have to also remember to update the repeated parts.
In fact, my experience has been that for good design much more important than DRY code is simply reducing global variables (including classes -- which implies that you will be doing a lot of dependency injection). But, again, you can take it too far.
I think my biggest piece of advice for beginners is to get a mentor and constantly ask for opinions about whether a design is good or not (and what about it is good/bad). It's not something you can learn from rules. It requires a considerable amount of experience.
Loose coupling (which implies getting rid of global state) is at least equally important, as is failing fast.
I was reviewing some code yesterday and the developer had a bunch of interfaces for everything so she could swap out implementations and she was very proud of how modular everything was. The problem was that all of her interfaces accepted a File object instead of a stream so we could change what the behavior is but not what it acts on which is really the opposite of what we needed.
I find that most developers who think that they can predict what kind of coupling they will need for future requirements tend to be wrong, too, no matter how experienced. Only backwards-facing refactoring really works at producing a good result, coupling-wise.
DRY is generally meant to be an optimisation towards reducing the cost of change by reducing the amount of duplicate code (repetitive, busy-body work) but it often backfires and increases the cost of change.
For example, say you are tasked with creating 15 widgets. During your first few sprints, every widget is the same, so your colleague berates you for creating them separately. You refactor them so that they all share a common implementation and continue.
3 weeks passes and features are being added left, right and centre. A point of contention within your team is whether to differentiate between 3 different types of widget with different features, or whether to use dependency injection. Development is slow because only one person can work on the widget code at a time, and the widget code has becoming increasingly complex as the number of callers invoking it have given it increasingly diverse requirements (note how looking at a function generally does not tell you which other functions call it, how they call it or why they call it).
The question you should ask yourself in these situations is, how similar will these widgets be in 3 months time? If it's likely that they will diverge significantly then perhaps the best solution would be for them to be completely differentiated now: tested separately, easily updatable/extendable without regression bugs on other widgets, able to be worked on by multiple engineers without merge conflicts, etc. The unfortunate reality is that once you have made a unit of code DRY it is both time consuming and difficult to do this and politically untenable so people typically avoid it.
When should you make your code DRY then? (1) You should make your interfaces DRY early on as this is a safe win without the problems of DRY implementations, (2) When you are very sure that the implementation of different units of code will not change in ways that do not create intellectual work of the following kind: "How do I avoid complecting these unrelated behaviours?"
Having said all of this, perhaps this is not beginner friendly. Perhaps beginners should just follow DRY always. At the very least they will get practice refactoring.
DRY should only be avoided if the refactored code ends up being much more complex than the duplicated code and it doesn't reduce the duplication all that much.
Pretty much every time I've seen DRY used to eliminate repetitions of three or more times it's been a good idea and 4 or more times it's been a no brainer.
>The question you should ask yourself in these situations is, how similar will these widgets be in 3 months time?
This is an awful approach. You won't be able to predict how similar they will be and you will likely create expensive to maintain architecture for projected scenarios which don't exist and neglect the architecture for scenarios that will.
I've seen many codebases littered with architectural changes that were put in place for presumptive future features and then never used. It happens way, way more often than those architectural changes actually being used.
> > The question you should ask yourself in these situations is,
> > how similar will these widgets be in 3 months time?
>
> This is an awful approach. You won't be able to predict how
> similar they will be and you will likely create expensive to
> maintain architecture for projected scenarios which don't
> exist and neglect the architecture for scenarios that will.
Keeping things specialised towards a single purpose even if it is redundant in the short-term makes code easier to grok, test, and update.
Making a change across 5 redundant functions is not time consuming work and is not easy to mess up. However making a change to a singular function which needs differentiated behaviour for one of its callees is error-prone as (1) it might force an interface change and (2) there is a non-zero probability of a regression error on the other four callees.
I'm not attempting to predict the future. I'm attempting to talk from first principles about real cognitive and communicative costs.
I started using a rule of "use the language's primitives first" to guide my use of DRY. There are many ways in which you can "reduce repetition" by introducing a generalization abstraction, something that in turn couples lots of code to a master definition: Generics, high arity functions, and inheritance are three such temptations. Doing that, in turn, accidentally models the whole problem before the spec is mature.
But if your discipline is "I'll use more built-in types and simple data structures, I'll keep more of the code inline, I'll let this class get bigger and I'll make some duplicated simple, fixed-function code paths," you don't get trapped by that. Sometimes I end up with the same simple math function duplicated across two different modules because I needed it for completely unrelated algorithms: That isn't pretty, but it's not a gross failure.
The opportunities to make a really useful abstraction for a novel problem often take time to ripen, and until that point you might as well assume that the code necessarily involves a "spaghetti" mindset. You don't want to pay the cost of reworking the abstractions unless you have to - it presents a huge hazard to maintenance and it means the attempt to generify failed.
But I think coders do get caught in a "clean code" mindset now where it needs to look both pretty and succinct, 100% of the time, and that leads to pushing the functionality out of the function body and into "ravioli" code.
I think Go is a good language, of the hot ones today, for learning this kind of engineering-centric focus because it actively discourages trying to whittle down the code to the succinct one-liner. Unfortunately by the time I encountered it I already had this figured out(perhaps years later than I should have).
e.g.
widgets/
simple/
index
test
advanced/
index
test
weird-as-hell/
index
test
company-x-wanted-different-features/
index
test
tophats/
index
test
This is way simpler and easier to work with than if people have created an `uber-widget` that has a configuration object and some dependency injection to handle all of its features because some overzealous engineer once decided that he was certain that all of the widgets would be able to be represented by a fully generalised `Widget`.
I'm obviously not saying that all cases of DRY are like this, but I am saying that there are far too many people that end up doing this because they make decisions to generalise code too early.
I find considering whether the repeated code is innately related, or in the same domain, or not, to be the best metric for determining whether I should refactor it to not repeat. It's adhering to the principle of least astonishment. Sometimes it should all stay separate. Sometimes a few pieces should be refactored to share code, and others left to repeat. Sometimes one set of the repeated code should share a function, and then the other set should share a different function, -even if that different function has the same implementation-, simply because the second set is so unrelated to the first. Sometimes they all should share the same code, because they're all related.
Breaking things into small, reusable pieces helps avoid this because the dependencies you create are on small, easily understood, easily replaced snippets, which tends to have a very clear context/domain. It's easy to prove that the abstracted function is correct, since it's so simple, and it's clear what domain(s) it involves, since again, so simple, and thus the problem with any given bit of functionality is almost assured to be the composition of those functions, i.e., your specific use of it in this one instance. You can still run into issues if you reuse some of those composed functions, however.
In general, the greater the complexity and domain specificity of the code you're trying not to repeat, the greater the danger you're shooting yourself in the foot.
FP is especially beneficial, as such, because the higher order functions that tend to be common are both pretty simple in their function, and pretty universal in their domain.
So for example if you have something that is dealing with registering your website, create a directory called register_website and all models, functions and controllers to do with that code go in there. The talk suggested deleting the controllers and models directories. He also said there was duplication but everything was more loosely coupled and you could always group things in deeper functional units later.
I'm not 100% brave enough to go for this structure quite yet but it certainly makes coming to a project pretty clear as you literally have a list of all the functions of the site where the code sits for each thing :-)
1. Automated tests are not the only way to write code with less bugs. The other two, currently less in fashion but IMHO very important, techniques are abstraction and assertion.
2. In my view, there are two kinds of automation - bad and good. Bad automation checks if humans did some process correctly. Good automation implements some process without human intervention. It's very easy to start working on bad automation rather than the good.
My personal rule of thumb is that checks for program correctness should all be automatic. Non-automated tests ought to be reserved for testing non-functional requirements (e.g. making sure the service is performant under load, reliability measures like failovers, etc.).
Actually, automated tests are best at ensuring the code doesn't have regressions. But writing bug-free code takes much more than just that; it needs thought-out architecture and interfaces, and with those you often can get decent code without automated tests, so no, it's not "the easiest way".
When doing TDD, often speed up development owing to the ability to get much quicker feedback on the code you just wrote.
Code without tests has a tendency to congeal owing to the massive uncertainty surrounding changing any particular part.
Oh, far from it. Tests are not documentation, documentation is documentation. Tests can be treated as a set of examples at best.
Bad tests not so much.
One class of assertions is type checking, but you can also have dynamic assertions while the code is running.
OTOH, abstractions are harder, but it only takes one person to invent a good abstraction that covers a huge class of bugs, and others can copy it. Examples of good abstractions that prevent many bugs are arrays with bounds checking, garbage collection, or optional data type.
Another example could be used linter or valgrind to prevent C-style bugs with pointers and allocation, instead of using a high-level enough language, such as Java, to avoid those in the first place.
It's absolutely a prerequisite.
Humans can model the behavior of small parts of a code base without the aid of automation, but once a code base grows beyond a certain size, you either need to have automation to have a handle on the code or let it congeal (which ends up accelerating the creation of technical debt).
There are bad tests just like there is bad code. All of the rules for good code apply to tests too.
What does it mean in this context?
Assertions are more general than tests, they test an expected state of application, not if specific case of input gives specific output. On the other hand, you can probably test more things that you can assert. So both have their place.
void function adjust_fan_control(uint8_t fan_speed) {
assert(0 != fan_speed);
... // some complicated code here
}
The second problem I have with them is in embedded systems. I'm heavily in favor of putting assertions (which are then omitted in prod builds, or instead of looping the LED blink routine they map to a soft reset if fail) inside functions in embedded systems. However, if I have very limited I/O, specifically, I don't have the ability to log when/where an assert was hit, all I know is that an assert was hit because the LED pattern goes to 'blink really fast like this indefinitely which means an assert failed'. If I have a lot of asserts, that doesn't really tell me anything.
On the other hand, if I have unit tests, and my mocks assert, and I can run the unit tests on a simulator or heavily instrumented system, then I know which assert failed in what function and who called that function. The point is I can't do that (or I don't know how) without the tests.
Edit: formatting.
A lot of the proposed advices are applicable only in certain fields, and not so much (or not at all) in different fields, whereas every field can go very deep (so it's a waste of time gaining knowledge you'll never use there). While a number of these advices are fairly generic, others definitely are not. A front end (client side) web developer won't benefit from having deep knowledge of data structures or the kernel, but he will benefit from knowing about Big O notation. Additionally, he will benefit lots from knowing how browsers work, and having deep knowledge of HTTP. Someone working on ETL processes all day will not benefit much from knowing about interpreters and compilers, but will benefit from knowing a lot about UTF-8. But also, he will benefit lots from knowing a large number of layer 7 transport protocols. A systems programmer (defined as "someone who programs close to the hardware") won't typically benefit much from knowing javascript or how browsers work, but will benefit lots from Big O notation, data structures and algorithms, and C. And so on.
I'm seeing a future in which we will be able to more distinctly treat separate specialisms within the field that we now call "software engineering" (or countless variations on it).
> A front end (client side) web developer won't benefit from having deep knowledge of data structures
but basic knowledge is still a must
I consider the basic knowledge of the "must" sections really a must for everyone. The depth might vary.
> Automated healing / rollback
> Don’t duplicate code
Everything else was spot on.
This is simple:
i = 0x5f3759df - ( i >> 1 );
famework.authService.login('user','password')
Simple is always easy to maintain.
Remember, as software developers, complexity is the enemy. It is the beast we want to slay.
if (str.length < 6) {
str = ' '.repeat(6-str.length) + str;
}
leftPad = require('left-pad')
leftPad(str, 6)
One thing that would be good to add is: If you implement a way to create/add something, implementing a way to delete/remove that something isn't optional.
As Lovecraft put it: 'Do not call up that which you cannot put down.'
This will result in a system that keeps proper track of stuff.
Nice post, by the way.
I would add "Software that does not ship is essentially useless."
To clarify, I mean that you can polish and try to perfect as much as you want, but at some point you have to get it into the hands of users to be actually useful.
Also, always, always, always, serialized in ISO-8601 format if it is getting turned into text.
Over the last 12 years or so I have constantly been bitten in the ass by UTF-8. It first started when I moved from the US to Europe and decided to help people make websites in their native language.
More recently I was developing a piece of python code to help me with a thing. Running into UTF-8 problems made me decide to do a deep dive into what it is and how it works. After a week of diving down a rabbit hole I am still none the wiser.
Blog post after blog post, video upon video, and it still doesn't click for me. Yes, I solved my problems, but in a black box type of way.
Can anyone recommend a good book or long source to help me with this?