Extreme Programming, a Reflection
blog.8thlight.com
blog.8thlight.com
I mean, pairing is a fine approach when training someone up on a codebase, but it tends to be much more effortful for the guy in the driving seat, while the guy looking over your shoulder is making small comments and doing researchy lookups. This makes it less than efficient when both people are at the same knowledge level. The extra eyes looking for bugs is debatable; the bugs are more thoroughly found with tests.
Test-first TDD is even less popular. Norvig vs Jeffries was enlightening - http://devgrind.com/2007/04/25/how-to-not-solve-a-sudoku/ .
Software does have a much heavier focus on testing than it used to, to the point that in many projects, everything is implemented twice - all features have two representations, one in the form of the implementation, another in the form of tests, and often with the lines of test code outnumbering the implementation.
But other things have suffered IMHO; making code easy to test tends to over-abstract it, making it more parameterized and exposing more implementation details of high-level abstractions.
APIs are often uglier with a lot more exposed symbols to handle the parameterization, with various bits and bobs asking for interfaces that only have one concrete implementation that can only be created by a factory, and you have to learn the knack of actually instantiating the useful bits anew for each library. I've got Java squarely in mind, of course, and I'm convinced better language design can solve the problem with less harm to the software.
If the goal in writing software is to reduce it to a set of pieces that plug together, I'd agree. But it's an arbitrary metric and not an absolute measure of quality, not by a long shot.
Note that I don't say "composable", because composability is something that needs to be designed in, and it isn't usually clear how to do it best until the third or so time around - the rule of threes.
Furthermore, I don't say reusable, because reusability is something that also needs to be designed for. In particular, reusability in such a way that software can evolve over time without breaking clients (reusers) of the abstraction demands a tight and constrained contract that is broadened carefully, while testable components demand broad and flexible contracts, otherwise not everything would be available to be tested.
Every part that testing has forced you to break out to be individually addressable is a part that you cannot remove in a refactoring that significantly changes the way a library solves its problem.
A library that has been broken into parts that are neither composable nor reusable is simply over-complex. Almost every extant Java library is like this!
Of course, if you just write end software in small teams, none of this is relevant to you. But it is crucial in library design, especially when client code is outside of your organization.
As for reuse, I like this quote: "Don't aim for reuse. Write small, independent components you can reason about, and the right pieces for reuse will fall out." Jessica Kerr @jessitron on Twitter
I don't think anyone disagrees that a "well designed and written" program of loosely coupled pieces has advantages over a monolithic one.[1] The debate is really over which is easier to do. And, the argument you are responding to is essentially, that it is easier to abstract out parts after you have done it in whole a few times. I know, personally, that that is a very compelling argument.
[1] Well, there probably is some debate on the feasibility of making things as loosely coupled as you would like. Back to the kernel debate, how many microkernels have survived with the device support of linux?
Software can easily consist of loosely coupled pieces as source code and be compiled and run as a single monolith with hardly any performance loss at runtime (versus coupled source code).
That is to say, that there is appeal to the "loosely coupled" dream of a software solution. Not just in source but in execution. However, there is the reality that this is very hard. The contention in this thread is that to think you can start in the loosely coupled set of parts is very ambitious.[1] It isn't that it is a bad goal. Just that it is akin to wanting to score well in a marathon without first running a few smaller races.
[1] Unless I am misrepresenting that, of course.
Trying to use the kernel as a template to guide all software development is not a great idea.
I have yet to see a prescriptive approach to this that works. About the best I've seen is the holistic iterative approach. First make something, then look to see where you can isolate changes and make them. Repeat. If this fits a model of TDD, it is new to me.
Programming speed is not the only consideration when it comes to shipping software. Squishing something together as rapidly as possible may shorten the programming time (and then only for smaller systems), but only at the cost of shoving the time into all the other phases, usually at a ratio greatly in excess of 1:1!
In other news, programmers are generally pretty bad as estimation, and this is probably related. I suspect the estimations for the "squeeze something together" part are pretty good overall, it's the rest that breaks down.
And again, to be clear, I'm not disagreeing that it's challenging. I'm saying that rather than being fundamentally challenging in a way that can never be made easier, it is a skill that can be learned. That makes for a very different cost/benefit set than a task that is fundamentally difficult. And, frankly, few developers are taking the time to learn it; far more are sneeringly dismissive at the skills that are required to learn this. Rather a shocking amount of our "structure" in programming is still just covering over cowboy programming with terms management can get behind. I think XP actually avoided this, but the average bastardization of XP is a thin patina of words over cowboy programming.
Now the major trick here is that this breaks down in categories that are effectively already solved. Which is why many of the examples are obnoxious to the point of unhelpful. If you know how to break something down to where it is loosely coupled parts already, I feel you should definitely do so.
Linux is well designed and you can learn from it, but in order to get value from that study you need to be a skilled C programmer and at the top of your game. Therefore it is a bad example for people who mainly use other languages.
In additon lets not forget that the SOLID principles, DRY, YAGNI and so on, are not hard and fast rules. Every extreme programmer will regularly violate those principles. The purpose of the principles is to guide your work, to make you see clearly what you are doing, so that when you violate a principle you do it for a good reason.
[1] Consider also this lovely thread: http://www.realworldtech.com/forum/?threadid=65915&curpostid...
You can do simple
things easily - and in particular, you can do things where
the information only passes in one direction quite easily,
but anythign else is much much harder, because there is
no "shared state" (by design). And in the absense of shared
state, you have a hell of a lot of problems trying to make
any decision that spans more than one entity in the
system.In the Linux kernel, you use printk to print messages on the console. That is modularity. There are device drivers. That is modularity. There is a range of loadable/configurable kernel modules for many things and these used to be more visible when more people would configure and build customized kernels. The Linux kernel has far more of these modules than earlier OSes that I used (TI DX10, UNIX 6th ed., Xenix, 3B2 Unix, SCO UNIX). The Linux kernel is linked into a monolithic binary that runs in kernel mode, but it is composed of many modules, some of which are integrated at link time and some of which are loaded dynamically (lsmod).
There is a good reason why the kernel is more monolithic than a business app, and that is that the kernel is doing a vastly different job at a vastly different layer of abstraction than a business app. You might also note that there are still lots of jobs for C programmers but most of them mention "embedded systems". That's what the Linux kernel is, a big featureful embedded system.
Perhaps some day someone will write a book on integration and cover all the different ways in which functionality can be integrated to produce an application. Most developers lean far too much on only one way of doing it, i.e. the link editor. For most apps, loosely coupled integration techniques are more valuable.
Consider, you can have a device driver that runs fine "on its own" but crashes when run with another driver loaded. This is almost canonically the opposite of loosely coupled modules.
But if instead of a monolith, you have a set of components with well defined interfaces that have simple semantics (that do not leak abstractions) -- whether or not these parts are re-usable in other contexts -- then you almost automatically have higher quality software:
* Easier to test means it will likely be better tested
* Well-defined interface and abstraction and a small implementation means that reviewing/correctness becomes easy. You only need to understand a small component to review it. "Obviously no bugs" rather than "no obvious bugs"
* Easier to split the work across developers
* Easier to comprehend the whole as a collection of its parts
The total number of lines of code, or even the total complexity may increase relatively to a monolithic design. But correctness becomes so much easier.
You mention refactoring, too, and IME, refactoring can be both easier or more difficult, depending on whether it is within a single small component or across multiple ones.
If you add architectural/design changes -- then it is night and day. A monolith will likely have to be rewritten to make an architectural change reliably.
A set of components can easily be split, for example, so that a few components are moved to run on a different system with a network protocol between them.
I think this highlights a deficiency in testing tools. It's quite hard to, for example, change the system time when running a test which often means that you have to abstract out that part from the method you're testing and pass it through as a parameter.
You shouldn't need to do that (in python you don't!).
Also, there is a lot of APIs out there with very real world effects and integrations that it is pretty cumbersome to build mock ups of. Most API providers also just don't.
Mocking what would happen, say, when a twitter oauth token expires, isn't as easy as it should be.
On the plus side, UI testing tools seem a lot better nowadays.
But yeah, there's a serious dearth in good testing tools and bad language design (cough Java) that ends up making code unreadable.
Passing the time into your function isn't necessarily a thankless chore, however. It's actually quite similar to strengthening your induction hypothesis when doing a proof. Now your function doesn't just claim to work correctly for one time (the implicit clock time) but for all times. This stronger claim (if true) makes it easier to reason about the code that relies on the function, including not only the testing code but also the rest of your application code.
I never bought into TDD, although I do write unit tests for most of the code where it makes sense.
No one has been able to show me an usable way to do TDD when coding native UIs, mobile OS, embedded systems or when using third party libraries not built with testing in mind.
Plus TDD makes very hard to properly design algorithms and data structures, that should beforehand be done at the whiteboard.
http://pragprog.com/book/jgade/test-driven-development-for-e...
I agree. However, I do not consider this a strike against testing; I consider it a strike against native UIs, mobile OSs, embedded systems, and third party libraries that don't support testing. You may not do TDD (I generally don't), you may not strive for 100% coverage, but testing is a fundamental aspect of serious software engineering, and anything that actively fights your attempts to test it is a big strike against that tech. I only use the ones that fight you that hard because there's unfortunately no competition, but it's still a disgrace. In 2013, testing ought to be a fundamental first-class concern of any new UI library, yet here we are.
To elaborate, I've tried pair programming myself and it was completely inefficient when we tried it. I'm not going to dismiss it entirely though, perhaps we approached it the wrong way. Personally I just need a bit of space before I can start focusing in-depth about certain problems.
This is also why I like to be well-prepared before attending team design decisions, because coming up with good ideas "right there and then" is difficult for me.
I think the trick is to use it when developers feel necessary to stay productive. No point having someone slogging away at something they find difficult and frustrating if a second pair of eyes and maybe some more specific knowledge of the area can help.
I pair on all production code at work with only two other guys. I've worked with them for the last year. Together, any combination of the three of us is easily twice as effective as the fastest in our team. Something about the rhythm of the session, alternating roles, support when tackling boring parts, and the camaraderie frees us up to just get stuff done.
But, we work in a very complex domain that, a year in and many seminars by product later, we only barely are starting to grasp, with a large difficult to grasp system, sometimes solving problems just outside our comfort zone. It's not just CRUD and forms. So, maybe pairing is the four wheel drive of the programming world: uses more gas on the highway, but depending on your terrain, it might be the most fuel efficient way to get across a mountain.
Or: I've tried Vim and my writing/editing speed halved. Or: I tried APL but it took me half a day to write one line of code. Or: I tried playing guitar and it sounded horrible.
I get the feeling that maybe the outcome would be different if you'd try doing it for a while longer. No guarantees, though.
What bothered me about how I saw pairing used was that people seemed to make blanket assumptions about it's benefits. Many times I saw people pairing up on trivially easy tasks. Seemed to me that pairing was a lot like everything else, it can be done well or poorly.
Pairing should be a naturally occurring process IMO; ie I don't know how to best accomplish a task or 'story', so I ask a team member w/expertise or experience to help point me in the right direction. If I need help beyond that, it becomes a pairing/knowledge-transfer exercise. I came to refer to it as "informal pairing". In general I tended to gravitate toward pairing on the exceptionally difficult tasks or ones that would have far reaching design implications.
When it works well, having two brains working together brings the benefit of different perspective and experience. It gives the opportunity to riff off each other's ideas. You spot issues with design and implementation earlier because having to communicate your ideas means working harder on them before you try to turn them into code.
I find it fun to work with someone else who is smart and engaged. It's magic when you become warmed up enough that things really start to flow. I've actually managed to get into flow before while working with someone else.
That said, it's pretty difficult to get right. I found it quite hard to let someone else see my process. If both sides aren't engaged in problem solving it can be really boring. I've also found that it takes me a while to figure out how to work productively with new people. The dynamic between any particular pair of people is a bit different. I think you need to build trust with your pair.
- Knowledge transfer. Learn new and better ways of working. IDE usage, short cuts, etc.
- On a complex piece of software is better to have two sets of eyes checking everything.
I'm somehow surprised that this even works well with pairing of programmer and hardware engineer, but that requires management that believes in their engineer's skills (rapid iteration and hardware tends to get very expensive very fast) and programmers that have meaningful insight into hardware.
On the other hand projects I work on are probably not very representative of anything as most of them are weird :)
I've seen more people extoll (and consultants sell) automated testing than folks actually use it, let alone TDD. As an idea, I get it, but the implementation still seems mixed.
And I'm in the programming industry because I enjoy programming, so would want to work in an environment which I enjoy - and in today's market, i have the luxury of picking my employer.
I'm sure not everyone feels the same way, but I suspect I'm not alone in that opinion.
Is the result better than with careful code reviews [of the critical pieces] (both after writing, but also short checks during development over a code listing and coffee)?
Intuitively, everything with coffee involved ought to be better! :-)
Better than some theoretically perfect practice of careful code reviews? I don't know. Better than code reviews as actually implemented everywhere I've worked? Yes. (In particular I find it's really hard to maintain the discipline of carefully going through each other's code when you know that most of the time you won't find anything)
That being said, I do miss solo work, because I could get into the zone, and even if I was going down the wrong path, it was ME doing it, master of my own domain. It feels like I'm alone on my boat sailing into uncharted waters, an adventure.
At its best, pairing feels like being part of a tactical response team, at its worst: like there is a machine that turns my brain cycles into money, and they let me keep some of the money at the end.
Edit: Rereading both comments, apparently pairing makes me wax with metaphors, like a... nah
"A complex system that works is invariably found to have evolved from a simple system that worked."
http://www.craiglarman.com/wiki/downloads/misc/history-of-it...
And before that for other activities: "Plans are nothing; planning is everything." - Dwight D. Eisenhower
He's one of the good guys, that's for sure - but every time I read his blog I keep waiting for that something new and it never comes.
Many companies are still run waterfall like, or without any kind of process.
When we bring in agile methodologies into the project, it starts slowly, but eventually everything is in place and everyone is doing it in an agile way (XP, SCRUM, whatever).
When the first project escalation arrives, or the deadlines are not possible to be achieved, the developers start slowly going back to the original way of working.
In the end you get mini-waterfall projects with a sprint duration, but the management puts agile on the project bullet points.
For most enterprises automated build, test, deploy (ie CI/CD) is the one missing tool and one absolutely necessary tool to capture and keep benefits of agile - it's capital.
And also for most enterprises you could lose 1/3 of the IT staff without noticing.
(Great Process + Average People) < (Average Process + Great People)
Although, one has this feeling of playing Quixote all the time.
From June 2011, this article covers four key success factors for the next 10 years of agile. Great thinking on software development.
There are definitely tasks in which I would avoid pairing - specifically those that are either very ill-defined (like a spike or bug hunt) or too easy (write some data transformations). However, the tasks that should result in a clean, well tested API with edge cases taken care of, tend to be higher quality while pairing in my experience.