Hacker News Reader

I have had a similar experience working with financial engineers and traders turned programmers (I am a programmer). They don't seem to value software the same way as programmers. They seem to view their ideas as the most important aspect of their work, and writing software is only a way of expressing their ideas. The problem with this is that if you don't feel that their are non purely functional requirements worthy of your time, pretty soon your code base will be impossible to work on.

Ive seen some bad stuff, including thousand line methods and production software whose source code is nowhere to be found. But the biggest forehead slap that I think I've encountered is when working on an automated trading system. I had just started working on the system when the original trader who had developed the algo quit. I saw him after work at a bar a couple weeks later and made some comment about how difficult it was to work on that codebase. He said that he made some of the trading logic purposefully difficult to understand so that it would be difficult for someone to read the code, get a job elsewhere, and replicate his work for a different company. Thats an insane way to write software. It assumes that all of the value is in the idea itself (and this algo was hardly rocket science by the way) rather than in a functional piece of software that is easy to modify and debug.

The other problem with it is that they don't appreciate that probably more than 90% of all code (including the code that touches their ideas) is for unsexy reporting purposes only. The domain expertise accounts for probably 1% to 5% of any business's code, and generally it's the easiest code to design, test, and change later on because it follows very tightly with well-worn practices in scientific computing. You might invent a great new trading strategy, but implementing it is still just the same old vector math and efficient algorithms stuff everyone's been doing for decades.

But with the logging, reporting, parameter management, data provenance, data resource management, etc., etc., the actual creative design of the code matters much more, and a failure to make it extensible is way more expensive to fix down the road than tweaking some O(n^2) scientific algorithm.

As an aside, this is one reason that I think algorithmic brain teasers are a really stupid thing for interviews or predicting positive business impact. With brainteasers, you are testing for rote memorization of a set of optimization techniques (data structure implementations) that maybe touches 1%-5% of your easiest-to-optimize-from-textbooks-or-wikipedia code, whereas the other 95% can almost always be assumed to rely on library implementations of these kinds of structures, and the high-level design of components, particularly optimizing it so that the addition or refactoring of features later is not expensive, is the dominant concern for adding actual business value.

It's just another status game like everything else. Programming labor is asserted to be a commodity item to the company, so that managers can justify lower wage, worse working conditions, etc., for programmers than for business domain experts who are supposedly less easily replaced. But the reality is that a reasonably smart programmer can probably pick up the domain expertise to a high level, even to the same level as a Ph.D. researcher, in a short time, whereas for some reason because of some kind of status-based mental block or lack of curiosity, the domain experts seem incapable of picking up legitimate programming skills.

When most of the code is for reporting, designing the reporting system is a huge priority. But when domain expertise carries political status, the importance of system design is often neglected in favor of campaigning for the supposed critical importance of the favored domain area.

Im not sure its a 90-10% split between the importance of infrastructure to business logic code in our case, but I don't think you necessarily need to make the distinction.

Something that I've run into a few times is when an FE will come up with a proof of concept that needs to be built out into something more robust, and it takes longer to build it out than to come up with the concept in the first place. This is often because the original POC is more or less a direct translation from pure math, while the real application needs to be worked into a multithreaded program for performance reasons. That often comes with significant changes to the way the business logic gets executed.

Software isn't merely a concrete implementation of pure mathematics. It is a logical system that bears some resemblance to math, but has a number of details (hierarchal memory model, network latency, etc...) that make it act quite differently. I think you can iterate on an idea faster and with fewer bugs if you treat it as a software problem from the get go rather than working on the idea and the software separately.

For example, I once had a frustrating argument with an FE about why a program he was working on wasn't working correctly. He thought I didn't understand the normal distribution and how the tails never quite reach zero. What he didn't understand is that the computer doesn't care what a normal distribution is supposed to be, if you try to deal with numbers on the order of 1e-100 in a float in c++, it might as well be zero.

A little harsh, but yes in general scientists don't write great code or use the right tooling. The reason is simple : the purpose of science is not to write code but to write papers.

That's not _at all_ to say this state of affairs is a good thing, but the practical reality is that scientists are judeged by academic papers, not code.

So, time taken learning how to code is time taken away from learning to write papers. Time taken designing code, testing it, documenting it, is time taken away from doing the research that goes into the paper. And so on.

Furthermore, most scientific code is actually scripting built on underlying frameworks. For example, plugging together a bunch of analytic tools to process some data. Or transforming data from one format to another before running it through a machine learning framework.

There _are_ scientists who write well documented, well designed framework code that is properly stored in a VCS and has release cycles, code review, etc etc. However, it is usually done in 'free time' - such as it is - and with limited resources.

There's also very little incentive to write code that's ultra-transportable. For the most part, I'm using it, for this one specific thing, and that thing alone.

I once refactored some code of mine to make it way more accessible and automated. It was, to be blunt, a complete waste of time.

I think that's a bit unfair to label all scientists like that. There are many of us that care about writing good code and working in version control. You just need to browse github to see many such people. There's not much incentive to actually do this in an academic environment, however, other than improving their own ability to improve and maintain their own codes. A well documented code available for everyone is usually well down the list of most scientist's priorities, for understandable reasons when you must publish papers.

Although I have quite complex programs I have written, which do benefit from software engineering methodologies, most of what I write are single-use scripts to automate tasks or try out ideas. These definitely aren't for public consumption and shouldn't be - they're more extensions of the thinking process than products of the research.

All generalities are unfair to some extent, it's just how generalities work, so I tend to read them as a quantified statement based on the number of people in the group that the person making the statement is likely to have interacted with. In this case, it seems as if the poster works with scientists on a regular basis, and that is his experience with those he has worked with. That being said, I'll pile on some of my own anecdata in just saying that in school I was split between physics and computer science and my initial approaches towards software development were certainly influenced by some of the thought patterns that the poster was expressing. I've come around on a lot of things, but software developers definitely have a different mindset about software because we develop it.

Yes, I'm also aware that I made a general statement about generalities, though I think this one is a mostly correct generality.

This is accurate. I am a good example of this phenomenon. Scientists don't know how to code in a way that is intelligible to someone else, and in academic shops, there's probably only going to be one person who writes/understands a program in the first place. This one person probably views coding purely as an icky means to an end, and his adviser thinks it's a waste of time if it even reaches his knowledge whatsoever. As soon as this person moves on, the software he was working on will be rendered nonfunctional.

Part of it is complete ignorance of standards and best practices; another part is a complete lack of documentation (too much time spent on non-science activity to justify making it). It's a problem because scientists can always make use of information technology, but they don't quite have the focus to prioritize it. Honestly, passing it off to non-science devs hasn't worked well in my experience, there's a knowledge gap that's really hard to get over.

I think the core of the problem is that coding and documentation are not considered part of the "science". Contrast it with writing papers, which is understood as communication with scientific peers, the community, and therefore is "science" -- and, of course, is one of the measures of scientific progress. Writing code is not. Anecdotally, writing papers about code is (on occasion), so maybe there is a way to convince the scientific community that the code is important too?

Some Scientists are used to describing something by stripping away all the irrelevant matrix and concisely stating the idea at issue. Maybe they write code that way - as little boilerplate as possible, just a few concise lines that capture the algorithm.

Or maybe they write hash, I don't know. But their approach isn't 'let me craft some good code', its 'let me describe my thesis' which will end up in a different place than any trained software professional.

I think a more fundamental problem is that coding and documentation aren't actually taught in a formalized way to many scientists, so you pick up the habits that worked for you at the time, and just carry those forward.

True, for the most part practicing scientists have not had a good exposure to proper software practices.

I don't think that a startup can solve their coding problems. Only scientists can solve their own coding problems by having the diligence and attention to detail that one would expect of a high school dropout with 2 weeks of a coding bootcamp under their belt.

What I mean by that is:

The startup I work at employees scientists, therefore the scientists are part of the startup, and it is our collective responsibility to create an atmosphere of good science.

Creating that atmosphere, in the face of years of habits and history, is a very difficult problem for a startup to solve.

Exactly. The root problem is that they're just not scientists.

"The code is bad therefore it's not science" is a questionable and even arrogant attitude.

In more cases than not, the software is doing something that was previously done by hand by some grad student or support staff(1). If those people made a mistake, then tough luck. Hopefully you'd catch it with a sanity check on the output. Were those people who relied on manual labor for centuries before the invention of modern computers not doing science? Absurd.

IMO, what's happening is that scientists are doing the same sort of validation that they always did -- placing no trust in the process (the code), and instead looking at checkpoints in the computation process and sanity checking the final output.

None of this is to say that there isn't room for improvement -- there is. But it's possible to do great science while writing shitty code. Hell, it's possible to do great Computer Science while writing shitty code!

(1) The vast majority of scientists programming are not, in fact, doing anything revolutionary with computation. It's just a more efficient way of doing what they used to do by hand. And those that are using code in an essential way typically (not always) have higher quality code.

Yes, it is a different culture. I really would like to learn how to help the scientists absorb the proper software culture.

>In my experience, scientists are like extremely junior developers with PhDs.

So from your experience, you are assuming every scientist disdains coding and does it poorly, and every programmer is a master at writing clean and working code.

Perhaps a scientist can teach you about the follies of anecdotal evidence.