Potato Programming
blog.glyph.im
blog.glyph.im
Now, in order to apply this ideas to a higher level of design, i.e communication between microservices, we need to apply the same principles. Reduce expensive calls by sending all data required to a function at the same time, let the backend do the work, precompute/prefetch data that will make work easier later...
Good advice in general. For me the highlight is that this approach can only be applied when we design and adapt our microservices to our computational needs instead of business. I still see a lot of gurus giving talks about splitting microservices in "business needs" and saying that anything else is a bad practice. Hold on, there is no silver bullet in engineering. Maybe the business logic is easy to understand, but this system design will make the system impossible (yes) to optimize in the future without a full re-write. And the same idea applies to low level design. Everything is a trade-off.
My approach is to design the public API following domain/business principles so it's easy to understand and implement. A la DDD and others if you want to call them something. But I want my backend to be flexible and designed around the REAL engineering requirements. If my `/api/foo/bar` is an async batch process that needs to be cached and my `/api/foo/baz` is a complex sync process, then `bar` and `baz` are two completely monsters. I don't want to have a `foo` service to handle both use cases. Having a `foo` service just because it's a business concept will cost you hours and money.
I have become very cautious when "gurus" speak about good/bad practice. My personal approach is to ask them which ultimate goal they intend to reach by that, and how their "good" practice is intended to reach that goal.
Note that I'm not asking for certainty at all, or even a high chance. Just a clear intention. More often than not, they cannot outline a path from their "good" practice to their end goal. If they can even name a goal, that is.
My favorite examples are splitting methods to reduce their size, and user story estimation (applies to both story points and man-days). No clear path to an end goal, not a single time so far.
> My favorite examples are splitting methods to reduce their size, and user story estimation (applies to both story points and man-days). No clear path to an end goal, not a single time so far.
Who here believes you asked that of someone and they couldn't give you an answer involving the words "simplicity", "readable", "understandable", etc?
Don't get me wrong, I would like to have a similar discussion with an actual subject expert, just most "gurus" aren't experts but just cargo-culters.
> Who here believes you asked that of someone and they couldn't give you an answer involving the words "simplicity", "readable", "understandable", etc?
I never claimed that they didn't give such an answer. What they did is give an answer that involved all these words, but without even the ability to define them except through the very same metrics that were supposed to measure them, i.e. circular reasoning.
The particular case of "readability" was the claim that breaking a long method into short ones increased readability, based on a vague idea (not a definition) of readability. I literally showed the code to a dev not involved in the project who was able to read and understand the long-method version better. While admitting the sample size of two (repeated that experiment once), I could give a very clear definition of readability.
I cannot give a similar example for "simplicity" and "understandable" since they were even less able to pinpoint those terms. I would have liked to argue at least on the basis of SOLID principles, but we never reached that point because method length (in lines) was more important to them.
And the reason is that it’s easier to write tests for functions with few branches without overlooking paths through the function. Same goes for reasoning, it’s easier to consider all paths through a function if there’s fewer.
I see number of lines as an imperfect proxy for complexity.
subjective reasons are still reasons, and they can still be valid reasons.
This is also a good way to check if their back-end code is free of race conditions, and after pointing out strange problems get a note from their developers that you should not send multiple concurrent requests to their REST API.
Been there, done that, with a huge vendor for, well, accounting software.
This is nice! This to me seems very sensible whether you have some fancy hardware doing P in parallel, or even just a plain old FOR loop - in the latter case it's the same length loop as the naive solution but you don't expose your procedure call mechanism to the loop. And it's a win on the calling side bc it'll help declutter the business logic there.
- don't operate on register-sized parts, operate on whole objects
- don't operate on single scalars, operate on whole arrays
- don't operate just on data, operate even on operators
what other members of this family exist? are worthwhile?
I am not a python dev, but I've been told that's because Numpy is largely written in C, no? Rather than solely because of an array/vector-focused "style" -- 5000% faster?
You need to adapt your style (in Python) to get these performance gains. In a language like Rust you can write normal, readable code and get the same performance, plus you don't incur the penalty of copying data between the language boundaries.
> NumPy. NumPy takes a similar approach: it cannot itself do anything to speed up Python’s fundamental interpreted execution speed3, but it can move the intensive numerical operations that it implements into operations on arrays, rather than operations on individual objects, whether numbers or not.
The move to operations on arrays would do nothing if the logic wasn't also happening in the C-written backend -- in fact numpy _did_ do something about Python's fundamental interpreted execution speed, it moved the logic to the C backend.
You point out that _also_ needed the move to array-based things to take full advantage of it, to minimize trips across the python/C boundary if I understand right.
But the gain is not simply from changing from operations on individual objects to operations on arrays. This is not really a demonstration of the array "style" having amazing performance advantages (5000%!) in general, generalizable to any platform as the OP does. It is instead in the numpy case specific to the enviroment of Python with C bindings, and wanting to minimize trips across the python/C boundary.
So yes and no. Moving to array-based operations removes the need for the for-loops in python that are slow. So you can do more on the C backend than in the slow Python frontend, by changing to an array-based style.
If you did this for single numbers, the boundary operations would be too much to make this useful. By moving the array into the backend we have a lot of computations in C, so it's useful to incur the penalty at the boundary.
Classic example: Batch upserting legacy data into new service.
Instead of looping and try to search for existing items before insertion, we could search first, then insert in batch!
Live processing requires progress updates for the user, and if it takes too long, is often relegated to batch jobs: other than that, it's equally easy or hard to optimize them for identical (absolute) time thresholds.
Batch jobs are similarly prone to races, starvation and other issues: you can't simply do a huge "upsert" without locking the full table, because it might lock other parts of the system, including other batch jobs accessing the same data.
If your performance goal is not achievable in time required for live processing (whatever you decide that to be), you just relegate to a background job and then it's the same as any other batch job.
Sure, this is a slight complexity on top, but it's such a common and well known pattern that it doesn't really add any practical complexity after you do it once.
It's a fallacy to think that it's any different.