Domain Knowledge or a Lack Thereof
jacquesmattheij.com
jacquesmattheij.com
Then these graduates go into the working world, and they typically can't help solve problems in economics, physics, applied mathematics, or medicine except by coding up some system to someone else's specs. The more dedicated ones make an effort to understand what they're doing. They exceedingly rare, gifted, and motivated ones might learn enough about an industry to make real contributions.
I hope that our field eventually evolves to the point that programming skill becomes an accessory. People will then learn to program just like they learn to write, and use the craft to help them solve their actual problems. The profession of programmers who only know how to program should disappear like the profession of scribes who only know how to write.
And some are almost proud to not know anything beyond their field.
Newsflash: You can be the best writer in the world, but what are you going to write about then if you know nothing besides writing?
Programmers often complain about managers knowing nothing about the field. But it's the same thing when the programmer doesn't know anything.
There is a lot of money in doing specialist applications. Most of these are made by people who know a lot about their field but only a little bit of programming.
Guess what? You're being micromanaged by MBA's because otherwise, despite all your l33t coding skills, you're f-ing useless.
Now that I'm over 80 credit hours in, I have to say that my Micro and Macroeconomics classes were two of the most interesting courses in my entire program.
Very easily identified early in school as the ones who say "I don't want to know all this history shit" or "I don't want to know all this writing shit"
In both cases, sitting there and watching was excruciating in that "holy crap this is awful, just make it stop for the sake of the poor guy" kind of way.
But you know what? Some of us got better at it - the ones who made an effort. This made me realize that we do not force enough liberal arts on engineers in school. In fact, I was prohibited from taking many art classes - I got no credit towards my degree for them because they were perceived as "soft". That is crazy.
I also think that reading and posting on places with high quality discussion like HN and reddit circa 2008 can really improve your communication skills.
Especially true if the business ever needs to write a technical proposal to win a contract. Very important to have engineers who can read and write then.
There are a couple of questionable assumptions baked into that statement, IMO. First, that a "liberal arts education" (by which I assume you mean earning a 4 year degree from a liberal arts university) definitely means you will come out able to read and write to a superior degree. Maybe if you major in English or Philosophy, otherwise, I wouldn't take that as a given.
Also, there seems to be an implicit assumption that the only way to gain a solid grasp of reading and writing well is through a liberal arts education (again, I'm assuming you mean a formal liberal arts education). I'm not sure I buy that. The best way to learn to write, IMO, is to write a lot, read a lot, and solicit feedback on your writing. None of that requires a formal liberal arts education.
Now if you aren't necessarily referring to formal / university education, then we probably don't disagree. I certainly agree with the point that having solid reading and writing skills is a valuable thing!
Ridiculous. I know this stereotype exists, but I don't know many programmers who actually hold that attitude. The "business shit" is intellectually easier and not at all uninteresting. The payoff in learning it is quite high: in a functional organization, you get more respect and money for a relatively small investment. There might be some clueless 23-year-olds out there who reflexively dislike "business" because of Marxist-academic indoctrination, but that's a rare and transient state.
If anything, the MBA micromanagers try to encourage this attitude by taking the "I will handle all of this" approach with the business side and the "big picture". They make it their turf by assertion, then wonder why the engineers under them seem not to give a shit.
Engineers tune out when they don't have any power or control. It's a survival mechanism. If you can't do anything about it, don't waste cycles on it.
We tend to prefer not working in teams, other people slow us down and cause confusion in our work.
But America's insistence on a "team" based work culture really hampers those like me. We can get 10x the amount of work done individually as we can as part of a team if left to our devices - but this is frowned upon, indeed looked at as a social "problem".
I * sincerely* blame MBA culture for this detrimental attitude.
I guess my question is, as a self-identified 'solitary worker', would you take it upon yourself to leave a project if it reached a point where team expansion was necessary? Is there any good way to manage someone like you to get you to contribute to a larger project, or if it a lost cause?
I much prefer the latter to the former since I don't have to work about anyone else makes changes to stuff without my knowledge but I'm still on the team, still met with the others to discuss what we are working on and how it impacts X or Y.
- no exposure to new skills and techniques. This particular employee is pretty fond of cut-and-paste, and we've been refactoring to fix a lot of that. As we refactor, the ground is sort of shrinking beneath his feet, because he isn't willing to work on other people's code. If you aren't willing to learn new techniques and read other people's code, your portion of the codebase is going to accumulate technical debt while the rest moves forward.
- no big-picture perspective. I love to stick my nose into every architectural discussion, and I think it's beneficial for us to plan big decisions as a team, to try and find the best solution. From experience, a siloed employee won't be interested in getting involved in these large-scale discussions, because they haven't kept up with the rest of the codebase, and they don't have an interest in it.
Essentially, having all your team hived off from each other, doing their own thing:
- gives you a very low bus factor. If one guy moves on, dies, whatever, suddenly you've got a dark area of your codebase with built up debt and no knowledge.
- reduces the cohesiveness of your design. At the best, the high level plan will be dictated by management or a single architecture, and each employee builds their part according to spec. This seems to reduce autonomy and job satisfaction, and it presumes none of the coders have any good ideas about architecture (wrong).
If you have techniques or suggestions for getting around these issues, I'd be glad to hear them. It is a big problem that some very good technical people aren't necessarily very social.
Working in a siloed environment doesn't mean not learning new skills or techniques. And I don't mean that you are only working on the same project/chunk of code the entire time. We would often switch projects so that what I'm working on today is completely different that last week and has been touched/edited/tweaked by just about everyone on the team at some point.
Having a Big Picture perspective is vital be being able to work individually. You have to know how your work will ultimately impact everyone else and how their will impact you.
I don't mean that everyone is working on completely separate projects but that Employee 1 might be working on a feature enhancement on version 4.5.3 while Employee 2 is working on a refactor of the back end code and I might be working on implementing a change control request on version 4.5.4.
Perhaps this is a better way to explain: everyone is working on the same Project but on individual and separate Tasks
Also, you can reply even during the cooling-off period by hitting the 'link' link.
Major feature enhancements in which multiple devs touch the same set of files.
There are probably ways around these issues, just some instances where I've had to sit next to someone and work out a bug/problem.
Not every project requires team expansion and teams vary in strength and productivity. But in general, yes I would leave if I were required to stand in front of 9 "team members" every morning and tell them my deeds and troubles. At this point, the challenges have been solved and the single hardest problem in the project is coordinating everyone. I get bored and frustrated at this point causing me to seek a more interesting project.
Software is fundamentally a creative discipline - creativity rarely occurs in a committee.
A 10 man team can certainly do more work then I can (in general). But if I'm unable to work properly in a 10 man team, I SHOULDN'T be on that team. Not all of us are effective in the traditional agile environment - give us a nook somewhere and let us solve problems independently. You'll get far more value out of us that way then adding us as the 11th member of that team.
I just want to point out that there are different aspects of "managing". There's tactical, command and control, administrivia type stuff, and there's "solving problems, acquiring resources and providing high level coordination" and then there's "strategic decision making" kind of management. One can certainly lack interest in one of those, but might find the others interesting if exposed to that part of things.
Personally, I consider most of what most managers do, to be "administrivia" which is of very low value, and I'd have no interest in ever doing that. But when a manager it tasked with actually identifying and solving problems, making strategic decisions, etc., that part can actually be as intellectually challenging as programming and can actually be fun in it's way.
It's not about control, some people just seem to be allergic to meetings, dealing with stakeholders, etc. It's a perfectly valid viewpoint to have: some people just want to write code.
That's absolutely true as well. I was largely that way when I was younger, but over time I've grown more interested in the other aspects of things.
I think the point is that you can do something about it - it just requires stepping out of the engineering midset for moment now and then. I've seen this fatalistic attitude frequently, both in software developers and "real" engineers (mechanical, electrical, etc). It's not healthy for the engineer or the organization, and I can't quite figure out where the attitude comes from.
With managers, you can't take the same approach. You cannot simply tell your boss that his managerial style is nukular buttfail. That gets you fired.
Engineers tune out when they don't have any power or control. It's a survival mechanism. If you can't do anything about it, don't waste cycles on it.
Yeah, I've definitely seen this. But, on the flip side, I've seen manager types who want the technical people to get move involved in understanding the business issues and develop more of an ability to contribute at that boundary (to the extent that such a thing actually exists) between the "business world" and the "IT world".
If you folks will indulge me a bit of (related) self-promotion, I've written two recent blog posts[1][2], touching on a specific technique and methodology that I see gaining some steam, which deals with operating at that "chasm" between the worlds. In the first part, I argue that, in the future to come, developers are going to need to learn more about the business world AND that managers are going to have to learn more about technology.
[1]: http://fogbeam.blogspot.com/2013/02/so-what-is-capability-ca...
[2]: http://fogbeam.blogspot.com/2013/01/why-capability-cases-are...
That changed a few years ago (can't remember the exact reason why) when I started learning about the industry I was working in (because it was something I wanted to know about) and saw my work getting better and coming back with fewer and fewer change requests and a couple of Oh, that was a great way of doing X. You get it!
Made a world of difference.
Too many people, especially worker bees like engineers and janitors, tend to focus only on their part of the job and not how it fits into the big picture.
Everybody "knows" how to write.
There's a difference between literacy and mastery.
Without fluency in the solution domain, the problem domain will simply remain that: a problem without a solution.
The Other Jacques's point was that as developers we often see lots of problem domains and that learning about them is useful.
I tell my customers that part of my job is becoming an "instant expert" in the part of the world they've called me in to help with.
The only subject I can think of with similar professional requirements would be law, where deep knowledge of the solution domain is required and quick acquisition of the problem domain is a predictor of success.
In many careers, it's the other way round - most physics I know taught themselves how to program, for example, but they're lacking some tools that would probably help them a lot (and physics are very close to CS). I've seen accountants do a lot of programming in Excel macros and Access too.
Doctors and Economists might have less interest in learning, but academy-minded ones will really appreciate the modeling they can do with the use of computers (I think I saw a medical paper where they reinvented some basic CS concept :) ).
Some of the domain training comes from practical experience - I started out working for the credit report sector, and then moved on to the insurance sector, so I've picked along some domain knowledge in those areas. Other colleagues have worked in logistics, and now know about their problems, etc...
Perhaps more shocking:
https://fliptomato.wordpress.com/2007/03/19/medical-research...
So coding skill is one dimension along which you have to be good. Domain knowledge is another dimension.
A third dimension is knowing the code base you're working in (which takes time, since most production systems tend to be big). To be a really effective developer, you need to be good along all these dimensions.
I always felt most of us on HN were well rounded individuals--interested in most areas of knowledge. Looking at the random articles that get upvoted along with ones on programming, it seems to coincide with that premise. Sure, we talk quite a bit about programming, but it's a just a tool for automating the repetitive work we do and to make ours and other's lifes easier. It's a means to an end and also something that happens to interest us at the same time.
Or the ones with highly uncreative managers who lack sufficient domain knowledge themselves.
If I could demand a second thing, it would be the basics of business. A large portion of software is directly about business, and almost all of it is somehow shaped by how business works. E.g., I'm volunteering at a non-profit this month, but the developer I sit next to is hip deep in a risk management problem.
That makes a lot of sense. To me is that because we are using "languages". With a language you write. And your problems are writing problems: syntax, commas, naming, verbs, etc etc.
What if we just get rid of "languages" ? We actually do not "write a skyscraper". We make a project for a skyscraper. Or we "draw" a skyscraper.
Are languages the best tool of choice to solve the problem in connection with the information (and its calculation)?
(not that I have a definite answer, just a question).
As an exercise, try to convey the idea "I'm diabetic and I need some insulin" via !language. Charades? Interpretive dance? Drawing?
And yes, we do "write a skyscraper". That's what blueprints are for.
In reality however I expect that most skyscraper projects involve many many pages of writing and formulas in addition to the drawings.
OTOH a computer at it's core is a machine that sequentially follows instructions. So any description of a computer program must be able to reduce to this.
Since around the dawn of computer (or at least in the 10+ years I have been involved) there have been attempts to remove the requirement for computer programming, but in reality nobody has come close to anything as powerful as a programming language.
The most effective progress has been made via abstraction, where you allow one instruction to be used to imply many. For example making a REST API call with 1 line implies a lot of operations on a lower level (TCP/IP etc).
I think this is fundamentally because any system that you propose to replace a programming language will either be:
1) Turing complete, in which case it could be classified as a programming language itself and will require knowledge of algorithms/abstractions etc to operate effectively. This is where you end up with flowchart driven development tools where the flowchart ends up literally resembling a plate of spaghetti for any reasonably complex system.
2) Not Turing complete, in which case you lose a lot of power as there are things the system will not allow you to express and thus lose the ability to deal with many edge cases. Since you are developing software and not just using it, the reason for the development activity in the first place is probably to deal with some edge case than an off the shelf program does not.
When I was younger I used to play with various "code free" game development tools and ran into the same fundamental problem.
For example, I have a shooting game and when the player is hit with some projectile he takes some amount of damage based on the projectile he was hit with. This was usually very easy to do.
BUT then say I want to add an "armour" powerup to the game. If a player with the armour powerup is hit then he takes damage but at a reduced rate that is proportional to his armour value. The armour value is also reduced based on itself and the armour piercing value of the projectile which hits. However if the armour piercing value of the projectile is below some value proportional to the armour and health values combined then no damage is taken at all. Oh and the armour powerup cannot stack with certain other powerups except under some specific circumstances.
That type of logic was impossible to express with these tools, or required finding creative ways to abuse the tooling in ways not intended which would in turn make other functionality more difficult/impossible.
I think it would be better to say that the new programmers would replace the old as writers replaced scribes. Because we do have people who only write, it's just not quite the same.
Indeed, that's why I made the comparison :) The new and old programmers he describes are not the same either.
I would have actually gone into teaching History if the United States school system was not as limited as it is these days in what one can teach students. I probably read and study history as much as I do about software development. However, I love Computer Science and have no regrets about going into it instead.
Any student that does not strive to be well-rounded in as many areas of knowledge as they can outside of their field is really missing out on life. Entrepreneurship in general is all about knowing what will make life better and it's hard to know what needs improving if one does not have experience in areas outside of their field.
A person does a great disservice to themselves when they choose to isolate their realm of knowledge to a narrow field, no matter which field it is.
It was only years later after I began delving into Linux and found a need to make something a bit more powerful than a bash script that I began developing an interest in programming again.
I'm sure there are many other capable and willing folks out there right now who feel the same way that I did. They want to learn but they don't see the point of it. Most courses at least don't really teach any real world applicability to what is being studied.
Now that I'm trying to get deeper into Machine Learning stuff, I find myself motivated to go pick up books on Statistics and Linear Algebra and dig in, when I had very little interest in that stuff before.
Items of zero value? A certain sci-fi author would beg to differ with you. Even someone that would have zero knowledge of anything else but would be mastering programming is still a technopriest. Even with zero knowledge about anything else but computers one still understand more about today's world than most people out there.
I'm not advocating refusing to learn other domains (I'm the kind of person who, when writing a book, decides to learn typography and typesetting software so I can typeset the book myself) but your cheap stab at young programmers (I'm 40) is uncalled for.
The world is going to shift to even more 0's and 1's. Many things are going to disappear and be replaced by 0's and 1's.
If there's one thing common to every single profession out there is that people need to read, write and communicate. And you know what? The tools allowing to read, write and communicate are written by the technopriests.
It's even more "scary" than that: the very infrastructure in place allowing people to communicate (like electricity and Internet) are, themselves, run using software. Not to mention very basic need like the entire infrastructure providing clean water to fullfill one of the most basic Maslow need.
There's no need for someone taking care of a network to understand what's inside the packets of information running through the network. All they need to know is that if there's no more Internet packets travelling worldwide, then there's no western civilization anymore.
The very foundation the entire society now relies on is software. If a worlwide EMP blast would render all computers useless tomorrow the world would probably turn to a mad max horror scenario in a few hours. No more water. No more electricity. No more food. Our logistics would be so screwed that "advanced" countries would fall into chaos. It's probable that in some places people would die after a few days because of the lack of drinkable water and it's very probable that in a few weeks many would die due to lack of food. Not to mention the nuclear facilities which would leak radioactivity left and right and break havoc.
Doctors aren't forced to understand how computers works to do their job. Most doctors are utterly clueless when it comes to technology: they're bound to live in fear of being hacked and not understanding what's going on when they connect to their online banking account and not understand why this or that mainstream article tell them to disable Java in their Mac. I've got doctor friends: they simply don't understand the world they're living in. We're already in the technopriest age. And they're asking me questions, lots of them.
The most computer-wise knowledgable doctor I know was thinking that the latest state of the art way to diagnose using computers was still to use an "expert system", filled by doctors. That's not how it works. These "experts" aren't able to process the amount of information a computer can process and can't find the correlation a computer software can find. Which is why now machines are trained with gigantic amount of information and suggesting exams and deceases to look for that are way more relevant than what doctors would suggest.
I think you got it totally backwards: it's a lot of the other professions that are going to disappear. Replaced by robots, machine learning algorithms, software and... The technopriests.
The ones still able to somehow understand the world they're living in and still able to write the world they're living in.
P.S: regarding mainstream economics I'd like to point out that Krugman, Nobel Prize winner, recently said that it wasn't a bad idea to mint a one trillion coin to deal with the U.S. debt SNAFU. Sure. Why not mint 26 of them and be done with the U.S. public debt!? That is the kind of economics most economics and most universities do teach worldwide: Keynesian economics. I happen to think Keynes was all wrong and keynesians have never been able to predict the consequences of their interventions more than a few months/years forward. There are economists from other schools who predicted 15 years before it happened, before the first euro was even circulating, that the euro would lead to: too many houses in spain, too many public servants in France, too many industries in Germany and who explained by which mechanism Greece and Spain would eventually default. Not a single Keynesian has ever been able to do a prediction that precise. That's all I need to know about "economics": that the mainstream is all wrong.
I do also think you put way too much faith in doctors: these people really aren't nearly half as expert as you think in their field. And they are the ones getting replaced today by sofware and robots. There's simply no way a surgeon can be as precise and have as many axis of rotation compared to today robotic arms (explained to me by a surgeon friend). There's also simply no way a doctor can read blood samples and other exams results and be able to correlate that with hundreds of thousands of other results and hence make more correct deduction as to what's going on.
If we eventually end up one day with true AI then everybody becomes meaningless because that day there ain't going to be doctors nor economists anymore.
Until that day programmers (even the one knowing nothing besides 0s and 1s and how 0s and 1s move around) are going to be more and more like the technopriests.
The truth lies somewhere down the middle, though I think you're a bit more extreme on your end of the scale than he is.
It's hard to take your views seriously when you describe your kind as "technopriests", and claim that people understanding nothing but computers "understand more about today's world than most people out there". This is a laughable claim.
I'm sorry, I can't get over this. Technopriest? This is the real world, not Snow Crash.
> "Many things are going to disappear and be replaced by 0's and 1's."
Yes, and they are being replaced by 0's and 1's in specific spaces where those doing the replacing must understand non-computer things about the space. Want to automate the entire traffic control system (LA just finished doing this)? Sorry, you have to know a great deal about transportation.
Want to automate food production? Sorry, you must know a great deal about both manufacturing and food. You can know all you want about computers, but unless you know these other topics you're useless. Well, you could be coding to a spec from someone who actually has a broad base of knowledge - a cog in the machine at that point.
The future belongs to people who have a deep understanding of computers and [insert relevant field]. Those who understand just computers have only a future pounding keyboards and not making decisions.
You know what, you're right - there will be "technopriests" in the future - but they will be the ones with a broad base of knowledge on top of technological mastery. They will not be the ones who understand tech and nothing else - those will be the digital equivalent of janitors.
Though they might fancy themselves technopriests while the world passes them by.
Everyone in our increasingly automated gift economy lives in fear of being cut off from the support of the layers of technology they rely upon for all their needs, all the way up your vaunted Maslow hierarchy.
You shouldn't romanticize this state of affairs, and you should not crow about the special role of the priestly class. In my opinion you should think soberly between now and your 50th birthday about how you can contribute to a global society that is both more stable and more technologically advanced, and try to forget about the futurism that was peddled to you ( and all of us) by Wired magazine in 1994.
I completely agree that the world is running on software right now, but the world isn't software ITSELF (well, not yet).
On the other hand, people being cracks in there field claiming that there field is the only one that counts are just arrogant. No mater if they a programmers, accountants, mech. engineers, doctors, car mechanics or what not. And yes, software written by people without the slightest idea of the purpose of said software tends to be not so good.
In the OPs specific example, the purpose of the network is to carry packets, and your job is to make sure packets are delivered. Whether they carry orders to launch nukes, or donations to charity isn't the purpose.
Any other centuries-old disciplines your knowledge of Big-O notation and compilers qualifies you to discredit? Oh, modern medicine? Good for you.
This guy takes the "hubris" as a virtue line way too seriously.
If they actually spent the trillion dollar coin, that would likely cause devaluation of the dollar; as long as the coin remains locked up in the treasury it's just an accounting trick to get around their own arbitrary rule and wouldn't cause devaluation, because everyone buying dollars knows it's just a stupid accounting trick and not actual money.
There is also the more generic field "Computational Science and Engineering" (CS&E) [1], which is primarily the development of methods and software that is applied in many specific disciplines. I work primarily on the library development side of CS&E, working with many applications to understand their needs, often finding ways to satisfy those needs with better software reuse and better ultimate flexibility, then implementing any components that belong in libraries. Additionally, we do basic research by recognizing unexploited common themes and designing new algorithms that will benefit existing and potential users.
There is currently a great deal of demand for Computational X, for CS&E through the postdoc level, and for CS&E in industry and national labs. It is recognized that a versatile practitioner of CS&E can effectively assist Computational X for many values of X, since there really is a lot in common. Once you've put in 10000 hours of CS&E, plus 1000 hours in a handful of disciplines, you should be able to get orientedy quickly in new applications.
At the faculty level, support for CS&E is highly variable. At some universities, X departments have the view that all their faculty should "X first, computation second", while the mathematics department wants to see theorems and isn't overly concerned with following through to see the method applied to obtain a new science or engineering result. When hiring committees say "we need CS&E, but some other department should have the CS&E faculty", it drives a lot of talent away from teaching roles (e.g., into industry), perpetuating the relative lack of early education in CS&E fundamentals. On the other hand, the trend is for universities to have more direct interest in CS&E (sometimes recognizing it with a dedicated department or inistitute). It helps that computation stimulates many interdisciplinary collaborations, leading to more diverse funding possibilities.
[1] We have a big meeting next week: http://www.siam.org/meetings/cse13/
Phase field models are mesoscale continuum (PDE) models based on the Cahn-Hilliard and Allen-Cahn equations, and are useful for things like predicting crystal grain growth and reorganization under strain, as well as fracture and many other uses. In phase-field modeling, the interesting physics is a consequence of the free energy functional, which is phenomenological and different for each material, so software should make it as easy as possible to turn specification of the free energy into an accurate and efficient massively parallel simulation. One such approach is outlined in this paper [1] from some colleagues. The resulting discrete systems are stiff and have positivity properties that must be preserved, so scalable implicit solvers (like multigrid) as well as methods for variational inequalities are necessary.
In molecular dynamics, there are many parallel scalability and expressivity challenges. Fast algorithms for long-range interactions (such as the fast multipole method) lead to less regular computational, but huge asymptotic (O(n) versus O(n^2)). Since better algorithms become more important as our hardware allows us to keep scaling up the problem sizes, Fast methods need to be applied in more places.
There is also a lot of activity in multiscale modeling and numerical homogenization, where explicit microscale models (often kinetic) are used to provide missing components of meso/macro-scale continuum models.
In a sense it's kind of sad: In building traditional software, understanding the customer is one of the hardest steps because you can't simply walk over and observe them going about their daily work, or ask them questions on a regular basis to understand their issues - you're separated from them by a sales process and probably some automated support tickets and a PR guy who won't let you have open discussion in public. People building tools for game development can have their customers sitting a few desks away!
One of the nicest things about the Agile fashion at the moment is that my customer (or a reasonable representative of him) is a couple of desks away and I really can go and ask him about what he wants. It's incredible how much time is saved by avoiding doing things he doesn't care about and how much happier he is with the end result.
My personal guess is that the reason there too is lack of domain knowledge. That many government projects are ran by handing out specs to the people tasked with implementing them.
Which, in hindsight, was a little misguided. There is so much value in learning and understanding the domain in which we are working in because it makes us so much more effective. The realisation came when I became frustrated in meetings where we would discuss ideas on financial models and I could not add any value to the discussion.
I then imagined working at a company like Google, let's say, doing stuff I wouldn't normally be interested in. Would I bother learning the domain knowledge? I couldn't say yes with 100% conviction.
I love programming but I also want to be effective in what I do. To me now, understanding the problems we are solving is probably just as important as writing maintainable code.
Now we should face three cases: 1) the programmer needs to learn the domain; 2) the domain expert needs to know how to develop; 3) something in between.
Most of developing is done in 1). There are some brilliant cases of 2). I once talked with a biologist doing some multi sequence alignment that told me that some routine in C was quite slow and was far better rewrite that routine from scratch in assembly. I was quite speechless.
In order to create better programming languages we could: 1) express domains in a better way. Actually, just be able to express domains would be really cool (and not so easy). 2) provide higher levels of abstraction.
I think it is just a people problem. Programmers tend to seclude themselves from their customers. Whole IT departments tend to seclude themselves from their users.
When you are looking for underwater welders for the off-shore industry, what is better, teaching a welder to dive or teaching a diver to weld?
(that's a serious question with a real answer).
It is much easier for a programmer to acquire the required domain knowledge than it is (usually) for a domain specialist to learn how to program. But you can only do so if you're prepared to 'dive in'.
I base that on the fact that the core value, the welds, will be higher-quality and that's more important than the diving execution.
The diving might be off, but it shouldn't be that hard to learn well enough to survive and be effective. Of course, I'm neither a diver nor a welder.
Did I win?
Yep. But a drowned welder is also not much ;)
https://www.commercialdivingacademy.com/underwater-welding.d...
That's just one program, but it appears diving is the first thing you learn.
Edit: If you look at it from the context of a normal welder, they have an understanding and control of their environment before they start welding. You don't realize it because you've been understanding your environment since you were born. To do underwater welding, you'd first have to understand how you operate in that work environment, and then they can teach you how to start welding in it.
My experience is that it generally has been managers shielding the programmers rather than the programmers shielding themselves.
I could imagine the right match of domain knowledge and the capability to learn how to develop in a non-tech guy and programming skills and the capability to learn the domain in a tech guy would be rather powerfull combination. And pretty rare one, I guess. At least for me, the actual programming part is... difficult. ;-)
On another point, this applies pretty much things like lean or six sigma, too. Both are quite usefull (or can be, that depends!), but people tend to see them as domain by itself. In practice, and that's only my personal opinion, this comes out short. If you add this to existing expertiece in a certain domain, then it works!
3. Create another EUP paradigm. It's been 34 years since VisiCalc came out, and despite it's success, I don't see a lot of effort to get end-users to program, especially in models, like the spreadsheet, where they don't know that's what they're doing. ;)
If you're interested in that sort of thing, ping me (see profile). My company does R&D in this space.
Given that, I'm surprised at the emphasis on 'reading' as the primary source of gaining domain knowledge. There are just 2-3 mentions of speaking/collaborating with the domain experts that are driving you to build the software itself. If you are lucky enough to work with acknowledged domain experts, spend time talking to them. Pick their brain. Two techniques I use: a) forward interesting articles to them with a question or an analogous concept in my domain b) resist the urge to show them a 'better' way. E.g.: In the HealthCare domain, I learnt people don't really want quicker or lesser clicks. They would prefer a set, stable workflow. If you cater to senior-citizens, keyboard shortcuts are less important than instructions in large font.
I still suck at engaging domain experts. So, I'd love to learn more techniques to hold an interesting conversation with domain experts. Ideas?
One of the first projects I worked on involved re-writing an entire admin system, both the technology and the underlying processes. One technique that worked well for the team, was to work-shadow; we would each spend time watching how a couple of people worked; compared notes and would go back and ask questions if we didn't understand it. Work shadowing doesn't have to be project related either; I visited nearly every department in our division of the over the years (1 every couple of months). I even managed to swing going out with the sales guys and visit clients; the clients were happy to spend an hour talking about their business and problems. Anyway, my experience with large corporates if that if you ask - people are generally happy to help you out - but I think many people just censure themselves from asking in the first place.
One other technique that I think works quite well is to get the strong end users seconded to beef up the QA team. Again, this doesn't work in all environments. But in the context of developing an internal system, it worked really well. For an external facing system, we did get a sales guy seconded onto the team too - which again really helped build up the knowledge of the whole team.
A variant on reading is podcasts and forums - I've used both of those to get to grips with whats going on in an industry.
For the software company, most of the above doesn't work, we're adapating Steve Blank's mantra of "get out of the building" and just trying to talk to as many people (especially customers) as we can. If they agree to talk to you, they're generally going to be helpful.
In addition, books can usually be found, on almost any topic, which are intended to bring a novice up to a basic level of conversancy. Reading one (or two) such books will make discussions with the experts much more productive.
Plus: it's fun to learn about new things.
>>"I can read a book faster than I can speak or hear words."
I disagree that this is an argument in favor of reading. Reading is easier but I think its not always as high in quality as collaborating.
>> "Reading one (or two) such books will make discussions with the experts much more productive."
Totally agree. I try my best to read as much with the end goal being picking the brains of my colleagues.
You've mistaken me for Jacques Mattheij.
Plus "Jacques" is an uncommon name in the anglosphere, apart from South Africa.
In water treatment, I learned about oxic/anoxic bacteria and how they're grown and recycled in the process to treat waste, how to write chemical metering logic in a PLC because I was the only one left on-site to do it, how motors are controlled with variable frequency drives, how proportional-integral-derivatives are used to prevent oscillation in feedback based controls.
In clinical trials management, I learned how double-blind studies are made, the process of randomization and dispensing drugs, medical coding and the life cycle of the trial.
In power systems simulation, I learned about three-phase power transmission, resistive and capacitive loads, active and reactive power, the challenges of hooking up a lot of solar panels our existing grids, how data centers are build and cooled and powered, microgrids, power meters, intelligent breakers and more.
I feel privileged that my skills at software development have exposed me to a variety of fields and that I know how several important processes in our word work. And I got to take home a nice salary while gaining that exposure. I do lament that, unless I pick a field and stick with it, I'll never get to work on the difficult programming problems that require a decade or more of experience and knowledge.
I find researchers (I am in academia) in both fields cross-talk too much. Computational/quantitative experts try to shoe-horn biological questions into their small corner of research area, while biologists don't care about computational efficiency, statistical robustness/efficiency, and the mathematical implication of their simple methods; no one cares about software quality. This is unfortunate. It is also unfortunate hiring committees do not look favorably towards multi-disciplinary researchers because one is not a trained geneticist/cancer researcher. I definitively do not encourage computer scientists go into support roles in academia; they are poorly paid and get little respect.
For technical domain knowledge i agree, but it also happens that this is often of personal interest to me anyway.
One thing that I have always had issues with when gathering requirements around something for an unfamiliar industry is that people who are in that industry will often neglect to mention things that are both important and non-obvious/counter-intuitive to someone unfamiliar with the industry.
This is probably because they are so familiar with the quirks of the way that the particular industry works that such things seem so obvious as to not need mentioning (and they probably don't see how it could possibly be any other way). Basically similar to how we are often amazed by assumptions some people have about the way that computers/software works.
The paradox of this though is that to deliver truly good software for a particular industry one must really know that industry as well as somebody who has years of experience in that area. At which point you start to feel that you could do their job too.
Another alternative is to have a very tight feedback loop with someone who has tons of domain knowledge and is the target user. Basically sit next to them as you develop the software and have them review every single change you make.
To create good solutions that are also innovative, I think that programmers only really need enough knowledge in that domain to be dangerous - to know more or less what happens but with enough gaps to not really know what's doable and what isn't. But, this knowledge is only useful if you can combine it with bits and pieces from several completely separate domains.
If innovation is the goal then I think there is a point where domain knowledge actually hurts you.
Of course there still needs to be at least one specialist to keep everything on track though - and I completely agree with you about having someone with deep knowledge on board. I think it should be someone who has been in that industry and will stay in that industry.
I can't see the point in programmers trying to become experts in areas that they're not interested in working in for the long haul. It's easier to learn (and actually apply) a little about a lot than a lot about a little.
Now, if you're designing a product to solve a problem in a particular domain you need the knowledge ahead of time, sure. But if you're a shit-hot coder and a fast learner, brought in to make things work, who cares?
I would agree with your original point though, that domain knowledge can be learned quickly by a good hacker. Which leads to a more interesting question; what techniques can we use to learn a new domain?
And then there are people like me who get called in because we're good at picking up large, existing, broken codebases quickly and somebody's in the sort of trouble that's best summed up as "Oh $&%£ we've got three months to deliver, the product's less than half-done and it's full of weird intermittent bugs". I'm finding it satisfying enough for now :)
My observation is if you can learn to program well, you can learn any field of knowledge. A programmer can learn consumer products easier than a brand manager can learn Python. Similarly a CS major can learn bond math easier than a Finance major can learn database design. (Yes, these are generalizations)
What's this mean? That bridging the Tech/Client divide is on the shoulder of the technologist. In addition to learning the latest frameworks, we have to learn our customers business. In addition to tech credentials, we should add theirs too.
Mastery of the solution domain and knowledge of various problem domains cannot be taught in a 3 or 4 year degree.
http://www.infed.org/biblio/communities_of_practice.htm
Basically, some organisations make their processes easily visible to most employees, and some don't. The ones that do can train people more quickly and improve their processes more quickly. Organisations that make 'peripheral participation' easy might be better customers for developers than those that don't. You might not have to make many sails or use a height gauge too often to 'see' what the issues are, just interview knowledgeable people carefully and cross reference the results with observation.
As a concrete example, the original author invested a considerable amount of (quite expensive) time in learning about sail making and lathe operating. In some of the 'business applications' I have to use daily, it is pretty obvious that the customer was a manager higher up and that the interface we have to use to populate the database that generates the wonderful reports has not really had much time spent on it!
There's a real limit to the amount of "truth" you can put into a specification, and at some point you have to trust the developer to "do the right thing". If they don't have domain knowledge in the industry they're writing for, they don't stand a chance ... and so your project fails even when everyone works hard and has good intentions.
Having to spin up on new domains also keeps work and life interesting.
Introduce a software engineer with an interest in the subject matter into that environment, and provided it isn't too big (like government), it might be self-correcting.
IMHO, with collaboration, it's more a matter of providing a hyper-rational process by which the purpose and priorities of the target audience(s) are focused and honed, and then producing the software as feedback, automation, and a usable compliment to their needs.
See http://en.m.wikipedia.org/wiki/Systems_analysis_and_design as a starting point.
In my experience of public sector software projects, the programmers seem to be kept as far away from the end users as possible.
I remember reading reports about some SAP installation process (written by idiots) full of stupid special terms such as "System Landscape" and "User Privileges" translated by an English teacher.)
Why we have this as a top news? Isn't it obvious that you must know the slang before you're trying to speak, let alone translate?
Well, most of coders use a slightly different strategy - jump right into IDE and pile it up, using all that containers and design patterns they have heard of (without understanding, why bother?) until it satisfies the manager (compiles) who know absolutely nothing (this is why everything must be statically typed). Then test it like a black-box - correct input produces correct output (never try incorrect or unexpected) and then ship and forget.)
We tried to go up the chain up to someone, anyone, who could explain how that case should be resolved but nobody could tell.
Why? Legalese. And just like most legalese documents: there's more than one interpretation possible because the very concept of legalese (based on thousands years old tradition) is inherently flawed and open to interpretation.
So, somewhere, some politicians, probably out of bikeshedding fashion, decides to vote an amendment exceedingly badly written and exceedingly complex. Then it becomes law. And law has to be followed. So that case has to be taken into account by the software but isn't (yet).
Everything is all and well until that specific case appear. The day it appears all bets are of. It's not working and nobody knows why. Nobody knows what is not working and why it's not working, nor how it should work. The law is bogus and the business case cannot be resolved without contradicting with the law(s).
Trying to determine this ended up with the programming team receiving a PDF with tens and tens of pages of legalese text that nobody up the chain (including lawyers) could explain to us. Probably that the politicians who voted these changes couldn't even themselves explain it.
We simply couldn't make the changes.
How did it end up? Someone very high up saying: "That's what should be done in this particular case" and then everybody covering its arse.
And there came the technopriests: fixing the production DB...
This is usually how it ends up.
I've been close to similar situations (but while creating something, not changing something that already exists), but usually, if nobody understands how this is to be done, the fact that you can't code it is second in importance.