Software Engineering at Google
swizec.com
swizec.com
Your tests are only as good as your mocks. They hide the true behavior of your system, drift away from reality, and take a lot of effort to maintain."
I've often felt guilty for leaning toward broader-scope integration tests, but I've long felt this too— that they're less effort to write and maintain and are much better value in terms of actually catching issues.
Not that unit tests don't have their place, but they're best for mostly-pure functions or classes where you're intentionally trying to cover a bunch of branches or corner cases.
Maybe the ulimate form of testing is something along the lines of Netflix's chaos monkey (now apparently an entire "simian army" suite of tools). Continuously test the real application, live in production, where it's not allowed to fail.
I think good unit test coverage is essential for good maintenance, but you can't cover _everything_ with unit tests. You definitely need integration and e2e tests to ensure bigger picture correctness. I have found that value one gets out of the different kinds of tests as you proceed up the pyramid get higher the closer to release you are, and are most valuable post release. During development they become obsolete too quickly due to changes in how things work. But locking in an already-released version to avoid regressions is great.
And that's fine, but I think it's much more valuable before making any functional changes at all, to impose some high level smoke tests that run the entire thing, and do a handful of common, end to end user flows.
The idea is that you test something at the very end of whatever you're working on, and ensure that it's identical to the original. You can then make larger changes to actually test the code properly while confirming that the overall system still works.
Super, super useful with ML things where running stuff can take a very looooonnnnnngggg time.
I'd probably phrase it as "Stubs and Mocks do not make sufficient tests", not that they necessarily make bad tests.
Yes, keeping mocks in sync with reality is hard but it should be up to the library owner to maintain either a good set of mocks or provide a fast enough library that integration tests are the better choice. This works in a company with uniform testing goals as a core value, not so we'll when dependencies and users have different testing philosophies.
So like— not mocking out the database interface with cooked responses, but instead using an actual instance of the database.
My corner of the world has ~100 people working on it and we do a mix of unit tests and integration tests. We definitely use mocks, but prefer fakes where possible. I really like our unit tests to poke and prod the corners of a given small section of code.
Our integration tests are just that, and they serve their purpose as well. But being able to stress all underlying parts of the system with all possible inputs at the integration test level is really really hard (if not impossible).
Many times, the way a system currently is can't push the corners of some helper class deep down, and you won't find those until someone makes some change in an intermediary layer. Making sure things are working at both the micro and macro level seems like the best way to go. But there is the obvious tradeoff of time it takes to do this.
Other teams we work with are write only integration tests. While it mostly works, I would say that it's harder for someone else to come into the code base and make changes. Unit tests help people understand how the code is expected to work today and how their changes may impact that. Integration tests won't always catch those kinds of things (plus tend to be way longer to run, which makes iteration time longer).
I don't think I have this experience first-hand, but I could certainly imagine a scenario where a well-intentioned person added a bunch of unit tests to a badly designed system, and they ultimately ended up just getting in the way and causing confusion when it was time to rethink it.
That's part of the reason why this post doesn't talk at all about user requirements, needs definitions, or user stories or anything like that. If this is what is "Software Engineering at Google" then maybe this shouldn't be seen as "best practice".
I have found the Google experience, as a user, to be decaying steadily over the past 5-6 years.
It is sin that is even worst than IE stagnation. ( Well I say that because IE is now irrelevant. If IE is still here my rage would have been even worst. )
The likes of Google and Facebook, as well as many startups inspired by them, have been an excellent demonstration of what commercial success looks like in a modern software business. In particular, they have (unfortunately, perhaps) demonstrated that quality really doesn't matter all that much in their target markets. In the early days, it turns out that users will forgive almost anything breaking if you have an attractive product. Later on, you can still break things often and almost completely ignore customer service as long as your lock-in effects are strong enough to keep most of your users anyway.
Even if you do need higher quality standards in your market, the lean startup view is to build anything you can and ship your MVP as quickly as possible. If and when you find product-market fit then you can pay off your "debts", potentially including big changes like rewriting your whole application or firing groups of early customers, because by that time you have the money and resources and user base to do these things without it hurting your bottom line too much. Again, from a business point of view, this strategy makes a lot of sense if you're operating in markets where these rules apply.
What I don't understand though is why so many people who are operating in markets where quality does matter seem to idolise big names like Facebook and Google and the people who work there. They have a high profile and they pay high salaries and their hiring processes are well known for being challenging, but I see little evidence that they effectively translate any of those apparent advantages into producing either better quality or more innovation than anyone else in the industry. If anything, given the enormous resources they have at their disposal, the opposite seems to be true.
Repeats a common error:
> QA takes a few hours or days to ensure everything works together as expected.
Quality Assurance starts at the front of the process: what do we do to ensure (i.e. how do we accomplish) that product does what it says, as reliably as we can reasonably make it? This could include processes like design reviews, code reviews, writing tests, coding conventions, even business processes.
Quality Control is a process of running tests and identifying things that don't do what they are supposed to (or weren't done as they were supposed to be, e.g. pushing straight to production).
As I was told years ago, and took to heart: "you can't test quality into your product." This doesn't mean "don't test", it means quality comes from the start, and tests identify failures.
No that is over engineering, and in violation of YAGNI (you ain't going to need it) an KISS (keep it simple stupid) principles. A lot of code my teams develops gets thrown away over time, as it is either no longer needed or replaced with an off the shelf tool. It is really hard to predict what code will survive AND becomes a performance bottleneck.
There is no point optimising prematurely. It is often best to tackle the problem when it occurs.
> When I first picked up Software Engineering at Google I thought it was another one of those FAANG books full of lessons that make no sense at human scale. I was surprised, the lessons apply to teams as small as 5.
> This is a "good shit stays" recap. The lessons that stick with you a few weeks after reading.
I'd be curious to hear from a real Google engineer.
Unseemly and oily.
I find these statements really weird when they're made by total strangers. They strike me as shallow.
The introduction is also weirdly phrased, as the process of SE doesn't start with writing code - that's putting the cart before the horse. SE starts with thinking about what you want to solve, and how you can and want to solve it. Coding should probably come after these processes.