Hacker News Reader

Comment by 9dev | Hacker News Reader

Oh look, another tiring craftsmanship debate that other disciplines long figured out!

A, say, physicist writing bad code could equally well be building a pergola for his garden. He doesn’t really know woodworking but god be damned if he couldn’t calculate the forces acting on the beams, and then add some screws - how hard can it be! And probably, he’ll even get the thing up, and it doesn’t look too bad even. Now get a carpenter over, and they will be horrified about all the things the scientist did unusually, did not account for, or just plain wrong. ”Wall Screws you still had around?? How could you not know you’d need structural screws for that?“, he will scream. However, the thing does roughly what the scientist supposed it should do. Until the winter, that is, when the wood expands due to humidity and cracks appear, and he finally needs a professional to fix the problems.

It’s the same story, really: It is a software engineer’s job to build quality software. A scientists job is to solve problems. There’s a clear boundary here, where the latter will deliver a concept to the former, who will eventually create a production-grade implementation off of that. Neither does a scientist have to build proper software, nor does a developer have to do cutting-edge research.

And all the words wasted on how one of them might be doing something badly is on he wrong path.

View on HN

I'm an ex-software developer/engineer and current scientist. In my experience TFA makes a good point, even though it's quite strawmanish.

Most scientific code is horrible from any sane software developer perspective. The quality is so bad that I think a huge proportion of published results are plain wrong due to bugs in the analysis. These apply to my much of my code as well.

But a lot of "software engineering" code is horrible too. Mostly because most popular technologies and "best practices" are just plain bad. Overenginering is a pandemic and has been a long time. Much of the roots is from the gilded age of enterprise Java. Totally misunderstood OOP. Byzantine layers of pointless abstraction. Rampant premature "web scale". Counterproductive bondage and discipline (yes, including much of static typing).

These have become a cargo cult in software development, and these cosmetic features are deemed as "quality code".

And this leaks into scientific programming. Even Python, arguably the language of science nowadays, and a "quick and dirty one", forces some of this cargo cult. Modules are needlessly complicated (e.g. relative and absolute imports are quite a mess), let alone the horrible packaging system. And the current trend to push typing.

That said, scientific code is getting slowly better, largely due to switch to public/open source code, and away from specialized hacks like MATLAB and R. Especially in more technical fields.

In software engineering OTOH things are IMHO getting worse.

I'll add that "code reuse" was pushed as an idea too heavily (at my university at least). I think that leads to a fair bit of the over engineering and unnecessary complexity.

I saw a comment later saying not to aim for "code reuse", but rather to "avoid code duplication" which made more sense.

I also agree that a lot of best practices are great if you are a massive company with unlimited resources, but they are just overkill and add complexity for smaller projects and teams.

Code reuse was indeed the hype, especially of OOP and inheritance. In practice it just made real code reuse a lot worse. The class hierarchies become so tightly coupled that any "reuse" requires a lot more boilerplate than the actual code to use.

I know that the idea is not just not writing more code, but have a "single point of truth". But the boilerplate leads to a situation where it's very inpractical and leads to a tightly coupled mess that's really hard to change.

What actually increased code reuse was duck typing (i.e. implicit interfaces), made popular by Python. But that's becoming verboten nowadays. And I don't think current structural typing systems are gonna reach the same level of reuse.

I basically agree (except I will take static typing over dynamic typing any day).

I work in computational chemistry, and scientists here don’t necessarily have problems actually coding (we’ve been doing it for 70+ years). But the “other stuff” is taking more and more time.

Before, you wrote Fortran, put the files on disks or whatever, and sent them around. Now, you need to know:

C++ and Python (and maybe Fortran too)

git and github and github actions, and packaging (pypi/conda/conda-forge I guess is the standard now).

Documentation? Sphinx I guess, although mixed codebases are still a pain. And where do I host the docs again?

Also better make sure it works on all three OSs.

Wait what is docker? Suppose I better learn make images. And they have to be hosted somewhere? Got to set that up too.

Oh wait again, Python 3.12 broke something. Have to fix that.

Does all of this make “better software”?. I’m coming around to the idea that maybe it doesn’t, at least in aggregate. Either way, scientists don’t have time to really learn all this AND the science they are doing. And it all changes every few years or so.

> Either way, scientists don’t have time to really learn all this AND the science they are doing. And it all changes every few years or so.

This is how I see it as well. I'm an immunologist, and it feels impossible for me to keep on top of my field of research and just about anything else. I don't have to produce quality software, but it seems difficult to keep your research cutting edge while maintaining software. It's hard enough keeping the research up to date!

> (except I will take static typing over dynamic typing any day)

Why? What exact typesystem do you prefer over Python's dynamic typing? A lot of its idioms, and probably e.g. NumPy/SciPy infrastructure would be about impossible with current static typesystems (look at the mess that is C++ scientific/ndarray libraries). Ditto for much of the autodiff and GPU stuff like pytorch.

Julia could perhaps get there, but the implementation has too many warts for it to take over.

It's more to do with developer ergonomics. Knowing what something is or returns (and having some guarantees about it) make it much easier to reason about code. Then, knowing (with guarantees) what a variable/object is, knowing right away what I can do with it.

> What exact typesystem do you prefer over Python's dynamic typing?

I just want to know what various objects are laying around in my code so I don't have to keep it in my head.

For example, if you are using a database library like psycopg, you run a query and execute it. What kind of object is returned from the query execution function? How do you check if it was null? You have to go looking at documentation, and often what you find is examples of what you can do. But those examples often don't include absolutely everything, so you have to go looking at the code itself. But library code in python is often pretty arcane.

With Fortran/C++/Rust/etc, you get all of that basically for free in your editor's autocomplete. And the compiler will check to make sure you didn't do anything truly dumb.

Dynamic typing also encourages some frustratingly bad habits. Lots of code out there will have functions that change return type depending on what arguments are provided. And occasionally someone will forget to return something in some branch and you won't know until you hit an edge case (during runtime) and suddenly you have a None coming from somewhere.

My projects tend to be in Python now, but I recently started a side project in C++ again and found it very refreshing.

> probably e.g. NumPy/SciPy infrastructure would be about impossible

Partly true, but maybe not as true as you think. I've been using the nlohmann JSON library for C++, and it is amazing. The code almost looks pythonic, and its type conversions are done automatically (although at runtime of course).

For example,

  for(const auto &[key, value] : some_json["x"].items())
    double d = value["other_key"];

EDIT: Also, I didn't realize how much I missed function overloading until I started the C++ project. Wow is that handy.

I find that "what can I do with the value" is a lot more important than "what the object is". Especially in scientific context this is usually "can I iterate this" "can I do arithmetic with this" etc. NumPy's broadcasting and support for "any" iterables is a great example of this. And very difficult to accomplish with at least the typesystems you listed.

E.g. going from a nested list in a JSON to a Eigen matrix is quite a pain even with (awesome) libraries like nlohmann/json.

I often have to resort to C++ for performance or libraries. I make a lot more bugs than in Python. And the bugs are often really tricky and often pass the typesystem (e.g. both nlohmann and Eigen have to really hack the templates and these cause a lot of edge cases).

Also serializing stuff in C++ is a real pain even with nlohmann/json because for god's sake there's still no reflection in C++ in 2024.

Of course you can abuse the power of dynamic typing. But you can just as well abuse anything else that's remotely powerful (see e.g. Boost).

"It’s the same story, really: It is a software engineer’s job to build quality software. A scientists job is to solve problems."

That's not the distinction. Good software engineers solve problems. That's what the paycheck is for. The distinction is whether code has to be maintained.

It's the scientist's job to solve a specific problem at a specific time. Who cares if the metaphorical wood rots next winter, the paper's been published.

It's _often_ the software engineer's job to build things that deliver business value over years, evolving and expanding requirements, in a development team, without grinding to a halt under the weight of accumulated complexity.

"Quality" software engineering is just heuristics for keeping the pace of change high over time without breaking things.

> It's the scientist's job to solve a specific problem at a specific time. Who cares if the metaphorical wood rots next winter, the paper's been published.

Sounds a little like cargo-culting than proper reproducible research. But this is a pest in academia definitely. Many papers do not provide all required data, all required model parameters etc. to get to the exact same result. Admittedly, they might nowadays need a software engineer to get that done.

Cargo-culting is about focusing on the process without fully understanding its purpose. Such as bureaucratic requirements for providing all data, software, parameters etc so that someone can reproduce exactly the same numbers with minimal effort.

Proper reproducible research is not like that. It's about providing sufficient details that other people in the field can extrapolate the rest. That they can use similar methods with similar data to achieve similar results.

Reproducing exactly the same result is not that valuable, as the "result" could be just an artifact of the specific data and specific methodology. Real validation depends on fully independent replications, with as little reuse of data and code as reasonably possible.

In software I have found getting the exact details and parameters very useful even if I don't intend to use them. Because when I try to do the same thing my own way and fail, then I can reference the original ones and gradually make my own version more and more similar to that, and see when it starts working. Or make their version more and more similar to mine and see when it breaks. This allows quickly easily identifying the critical difference.

If we cannot reproduce research results, due to missing the data, parameters, or code, or whatever else, is it not cargo culting, to take that research result and blindly believe it and build on top of it? If I remember correctly, Feynman stated in the same video, that one should actually reconstruct or reproduce the experimental results we rely on.

> Who cares if the metaphorical wood rots next winter, the paper's been published.

Isn't this why the replication crisis was able to be kept hidden for so long?

Not as far as I know. Much more problematic was the fact that replication is very hard to publish. That's because either you more or less confirm previous findings and therefore contribute little to the scientific record (or so reviewers seem to think), or your findings counter the original results and now it's on you to explain the discrepancy. Even if you satisfy the reviewers that you're right, they may not consider your result of sufficient caliber to accept for publication in this particular venue [1].

[1] I've seen this happen to a paper that proved that a theoretical framework for constructing proofs about RFID protocols was neither sound nor complete.

It is very true people should stick to the domain they know because otherwise they will have higher than average chance to f up.

But that 'clear boundary' thing is a naive bollocks! No such thing!

Both domain experts need to understand things beyond this imaginary and when precisely drawn then highly arbitrary boundary that is more like a gradient than a line normally (also not something relevant in a final good product).

Teams consisting members unwilling to wander into foreign territory and expecting to be fed and deliver over a strict boundary will do horrible things!

(we are not even trained this way btw., professions have quite a bit of overlap and we learn matters others are expected to take care of in practice, and in case of sofware engineering - apart from the most notorious ignorants no one wants to work with - it is pratically impossible avoiding to learn a big chunk of a foreign profession on the fly for delivering a good product, heavy science is no exception!)

Perhaps I phrased this badly. My point wasn’t a clear boundary between professions, because you are right, that is difficult to impossible to draw. However, there’s a clear boundary between the goals of the code written by scientists vs. software engineers. Where a scientist aims to prove something, a software engineer builds code to produce business value. Both are trained very differently towards these goals.

This probably highlights why so many software engineers in scientific R&D are "bad" in the strictest sense: they often don't understand that business value doesn't come from production-perfect solutions, but instead 100% of the business value comes from doing stuff fast, good enough, and understandable (so that researchers, not software engineers, can read it in a paper and iterate off of it).

I still disagree a bit. Their goals are the same: producing a product that fulfills the intended purpose and brings on value (financial or else or mixed). They may bring in their specialty learned beforehand (or even during) but this distribution of labour is not a goal but tool in reaching the common goal.

Also no such as a software engineer is trained to seeks business value while a scientist seeks proof with clear cut separation, not at all.

> It is a software engineer’s job to build quality software.

It’s our job to deliver value to the business at a rapid and maintainable way. Rapid changes are often worth more to the business than maintainability, even over a period of many years. In some cases you could put a non software engineer down and let them build something with ChatGPT and it would work perfectly fine for the next 5-10 years because it’s focused on something particular, doesn’t change much and lives in isolation from the greater IT landscape. In other cases all your points are extremely valid.

That being said we also work in an industry where a lot of “best practices” often turn out to be anti-patterns over a period of a decade. OOP is good in theory, and I know why we still teach it in academia, but in practice it very often lead to giant messes of complexity that nobody really knows how works. Not because the theory is wrong, but because people write code on Thursday afternoons after a week of no sleep and a day of terrible meetings.

After a few decades in the industry, what I personally prefer isn’t any particular approach. No, what I prefer is that things are build in isolated services, so that they only handle their particular business related responsibilities. This way, you can always alter things in isolation and focus on improving code that needs it. It also means that some things can be build terrible, and be just fine, because the quality of the tiny service doesn’t really get to “matter” over its life time.

I personally write code that follows most of our industry’s common best practices. Because it’s frankly faster once you get used to it, but I’ve seen really shitty spaghetti code perform its function perfectly and never need alteration in the 5-10 years it needed to live before being replaced.

Research software engineering is a hybrid field and fixed roles do not work well. It requires one having a grasp (at some level) of a lot of different stuff, eg software engineerin, the relevant scientific theories, statistics _and_, quite importantly, the culture of scientific practices in a field, in order to make something good. So it gathers a lot of people who, no matter where they started from, often have to sort of converge by learning stuff outside their own discipline.

The problem is not that "physicists write code". Scientists end up learning a lot of stuff and getting good at it (software engineers the same). The problem is that writing software is often left as a job for the occasional phd, postdoc or research assistant, ie people with temporary positions, and in general to people who see building software as a side duty at best, annoyance at worst. This results in no generational knowledge building, being hard to find mentors, less learning and reflecting on practices on how to build software, rediscovering the wheel constantly, too much effort put into . It is not that one graduates as "software engineer" or as a "scientist" and then they know the best practices and everything of their respective fields. People get to learn stuff. Software engineering practices should be in the culture of scientific software building, not simply carrying them from the software engineering world to science, but adapting them and taking it each own idiosyncrasies as a field.

I agree, a lot of unmaintained code is due to turnover of staff and few considering the software as an important output in its own right. But I don't think it would be that hard to find grad students/postdocs who would care about code, if that were something the field properly incentivized. Not only does software contribution not check the right boxes for career progression, it is also often looked down on.

I find this especially laughable in biology... I've seen some (faculty) PhD committee members object to the student having a thesis chapter related to software contributions, because this is not "intellectual". But they are perfectly fine with one of the chapters being a wet lab paper where the student was a 3rd author who contributed purely through helping run experiments designed by the 1st author (e.g. handling mice, pipetting shit). There are PIs that simultaneously hold these two views, which to me signals a real misunderstanding of the challenges in and importance of writing decent software.

The first distinction here should be what kind of "code" are we talking about. As far as I see there are two main possibilities:

  1. code for performing a scientific simulation or analysis (a script).
  2. code for solving a specific problem generally (a program).

There are different "best practices" for the two situations above. And the article primarily talks about applying "best practices" from the 2nd scenario to the 1st. Of course they don't apply.

>Neither does a scientist have to build proper software, nor does a developer have to do cutting-edge research.

Sounds like the mission of Research Software Engineers (https://society-rse.org/).

I work as a software engineer (with a PhD in a field that is not CS) in a research setting, and there is a give and take. 50% of my job is reading, understanding, and adapting very bad code from non-software engineers into a production system. But another 50% is binning the overwrought inflexible code written by my software engineer predecessors in order to do all that more quickly than refactoring would allow.

In a research setting, in my opinion, MVP is king. Researchers seem to usually not produce viable long-term solutions. But software engineers do as well by virtue of not being domain experts (how could they be, they would need a PhD to understand the research domain!) and being unable to test 100% of the assumptions underlying the software themselves. Which is why it can help to have someone in between who is a domain expert, but knows just enough software engineering to produce production-good-enough code.

This might work for CERN. But a great deal of science is done by small teams who don't have a professional programmer available. Basically all of the social sciences, for a start; a lot of genetics too.

If you cannot model using decent code is it worth writing models at all? What if bugs mean the model is simply wrong?

It has consequences too. There has been a lot of argument about how much impact the poor code quality of the Imperial college covid epidemiology model (which was the basis of British government policy during the pandemic) had on its accuracy. I do not know how bad it was, but it cannot be good the code was bad.

One problem is that it's really hard to tell when you've just written bad code, which is also a problem for people whose job title is software developer, not just people who do it as a small part of their overall work.

Some genes have been renamed because Excel interprets the old names as dates. The people who put all their genetic analysis into Excel had no reason to expect that, just as the people writing Excel itself weren't expecting the app to be used like this.

Yeah but, in your example. this is a bookkeeping issue that, while frustrating and time consuming and costly, is just that. It’s not like a gas line in Manhattan blew up because someone in Toledo hit C-v in Excel. The scientists swapped around excel files and imported stuff without checking, which then was fed into other systems. A clusterfuck but one that is a daily occurrence at, minimally, every major non-tech company. It eventually gets unfucked with human labor or it simply wasn’t important in the first place. Exact same thing happens in research and academia.

Point being, unknown unknowns are just that. But most unknowns are known and can be programmed defensively against for most serious use cases. All major fields are like this—like, you can hook up a car battery to light a menthol tank to boil two cups of water… or we can use a kettle. Perhaps for a brief point in time, due to our ignorance or just history, people lit containers of menthol on fire like it was sane, but that doesn’t mean it was, or is.

> I do not know how bad it was, but it cannot be good the code was bad.

I do know because I reviewed the code and its issue tracker extensively. I then wrote an article summarizing its problems that went viral and melted the server hosting it.

The Imperial College code wasn't merely "bad". It was unusable. It produced what were effectively random numbers distributed in a way that looked right to the people who wrote it (in epidemiology there is no actual validation of models, reinforcing the researcher's prior expectations is considered validation instead). The government accepted the resulting predictions at face value because they looked scientific.

In the private sector this behavior would have resulted in severe liability. ICL's code was similar to the Toyota engine control code.

A selection of bug types in that codebase: buffer overflows, heap corruption, race conditions, typos in PRNG constants, extreme sensitivity to what exact CPU it was run on, and so on. The results changed completely between versions for no scientific reason. The bugs mattered a lot: the variation in computed bed demand between runs was larger than the entire UK emergency hospital building program, just due to bugs.

The program was originally a 15,000 line C file where most variables had single letter names and were in global scope. The results were predictable. In one case they'd attempted to hand write a Fischer-Yates shuffle (very easy, I used to use it as an interview question), but because their coding style was so poor they got confused about what variable 'k' contained and ended up replacing the contents of an array meant to contain people's ages with random junk from the heap.

There were tests! But commented out, because you can't test a codebase that overrun with non-determinism bugs.

The biggest problem was the attitudes it revealed within academia. Institutionalized arrogance and stupidity ruled the day. Bug reports were blown off by saying that they didn't matter because the "scientists" just ran their simulation lots of times and took the average. Professional programmers who pointed out bugs were told they had no right to comment because they weren't experts. ICL administration claimed all criticism was "ideological" or was irrelevant because "epidemiology isn't a subfield of computer science". Others were told that they shouldn't raise the alarm, because otherwise scientists would just stop showing their code for peer review. Someone claimed the results must have been correct because bugs in C programs always cause crashes and the model didn't crash. One academic even argued it was the fault of the software industry, because C doesn't come with "warning labels"!

The worst was the culture of lying it exposed. The idea you can fix software bugs by just running the program several times is obviously wrong, but later it turned out that their simulation was so slow they didn't even bother doing that! It had been run once. They were simultaneously claiming determinism bugs didn't matter whilst also fixing them. They claimed the software had been peer reviewed when it never had been. The problems spanned institutions and weren't specific to ICL, as academics from other universities stood up to defend them. The coup de grace: ICL found an academic at Cambridge who issued a "code check" claiming in its abstract that in fact he'd run the model and got the same results, so there were no reproducibility problems. The BBC and others ran with it, saying the whole thing was just a storm in a teacup and actually there weren't any problems. In reality the code check report went on to admit that every single number the author had got was different to those in Report 9, including some differences of up to 25%! This was considered a "replication" by the author because the shape of the resulting graph was similar.

That's ignoring all the deep scientific problems with the work. Even if the code had been correct it wouldn't have yielded predictions that came close to reality.

Outside of computer science I don't believe science can be trusted when software gets involved. The ICL model had been hacked on for over a decade. Nobody had noticed or fixed the problems in that time, and when they were spotted by outsiders, academia and their friends in the media collectively closed ranks to protect Prof Ferguson. Academia has no procedures or conventions in place to ensure software is correct. To this day, nothing was ever done and no fault was ever admitted. There was a successful coverup and that was the end of it.

Again: in the private sector this kind of behavior would yield liabilities in the tens of millions of dollars range, if not worse.

Eek! That makes the worst code I've ever seen, seem good in comparison.

Aside from the code quality, his models have never been close to accurate on anything.

Right. That's not unique to Ferguson, epidemiology doesn't understand respiratory virus dynamics and doesn't seem particularly curious to learn anymore (I read papers from the 80s which were very different and much more curious than modern papers, not sure though it that's indicative of a trend or just small sample size).

Other models I checked didn't have the same software quality issues though. They tended to use R rather than C and be much simpler. None of them produced correct predictions either, and there were often serious issues of basic scientific validity too, but at least the code didn't contain any obvious bugs.

Wow. It's pretty unbelievable. It there a place where I can read the whole article?

The one I found the funniest/crziest is "Bug reports were blown off by saying that they didn't matter because the "scientists" just ran their simulation lots of times and took the average", because this is exactly how some scientists I know think.

This thinking is not limited to software. My father was by trade involved in building experimental apparatus ("hardware") for scientific experiments. Often they were designed by scientists themselves. He told me about absurd contraptions which never could measure what they were intended to measure, and extreme reluctance/defensiveness/arrogance he often met when trying to report it and give some feedback...

Yeah confusion between simulation and reality can be observed all over the place. Multiple runs can be needed if you're doing measurements of the natural world, but for a simulation that doesn't make sense (you can do Monte Carlo style stuff, but that's still replicable).

You could see the lines being blurred in other ways. Outputs of simulations would be referred to as "findings", for example, or referenced in ways that implied empirical observation without it being clear where they came from unless you carefully checked citations.

Here are some of the articles I wrote about what happened (under a pseudonym)

https://dailysceptic.org/2020/05/06/code-review-of-fergusons...

https://dailysceptic.org/2020/05/09/second-analysis-of-fergu...

https://dailysceptic.org/2020/06/11/how-replicable-is-the-im...

After that people started sending me non-Imperial models to look at, which had some similar problems:

https://dailysceptic.org/2020/08/08/schools-paper/

I don't write for that website anymore, by the way. Back then it was called Lockdown Sceptics and was basically the only forum that would publish any criticism of COVID science. Nowadays it's evolved to be a more general news site.

Thank you! This is very interesting.

> Outputs of simulations would be referred to as "findings"

Yeah, a recent brouhaha about creating (!) a traversable wormhole in a quantum computer comes to mind...

Given Ferguson's track record of being out by orders of magnitude on absolutely everything beforehand, it almost seems like he was chosen to give an over the top estimate.

It is possible. It is certainly common to pick the expert who says what you want them to - and to get rid of experts who say the "wrong thing" (e.g. the dismissal of UK govt drug policy advisor David Nutt).

It would be comforting to believe that because it'd mean there were other epidemiologists who were right but ignored. Go read the works by his counterparts though, and they're all out by similar orders of magnitude.

How are people supposed to do science without running statistical models?

This is asked in good faith of course, but that question really gets to the heart of what's been corrupting science.

Statistical techniques can be very useful (ChatGPT!) but they aren't by themselves science. Science is about building a theoretical understanding of the natural world, where that theory can be expressed in precise language and used to produce new and novel hypotheses.

A big part of why so much science doesn't replicate is that parts of academia have lost sight of that. Downloading government datasets and regressing them against each other isn't actually science even though it's an easy way to get published papers, because it doesn't yield a theoretical understanding of the domain. It often doesn't even let you show causality, let alone the mechanisms behind that causality.

If you look at epidemiology, part of why it's lost its way is that it's become dominated by what the media calls "mathematicians"; on HN we'd call them data scientists. Their papers are essentially devoid of theorizing beyond trivial everyday understandings of disease (people get sick and infect each other). Thousands of papers propose new models which are just a simple equation overfitted to a tiny dataset, often just a single city or country. The model's predictions never work but this doesn't invalidate any hypothesis because there weren't any to begin with.

How do you even make progress in a field if there's nothing to be refuted or refined? You can fit curves forever and get nowhere.

In psychology this problem has at least been recognized. "A problem in theory" discusses it:

https://www.nature.com/articles/s41562-018-0522-1

Right, statistical models are not sufficient for science. I agree. But they are necessary. So I recur to my original question.

Most teams at CERN don't have a professional programmer available either. In a few of larger projects (those with a few hundred active contributors) there might be one or two more tech savvy people who profile the code regularly and fix the memory leaks. But few (if any) are professional programmers: most contributors are graduate students with no background in programming.

And this is scary. At least with "high-energy experiments" (like the one that discover Higgs) in colliders, a lot depends on so-called triggers, which dismiss 99.9% of information produced in a collision "on the spot", so that this information is never recorded and analyzed.

They have to: there is way too much information produced. So the triggers try to identify "trivial" events and dismiss them immediately, relaying only the ones that are may be somewhat unusual/unexpected.

Essentially, the triggers are computers with highly specialized programs. Very smart people work on this, and supposedly they figure out problems with triggers before they affect the results of experiments...

The triggers are the most fun part of the experiments!

The composition of teams working on triggers might be a bit of an exception in the "engineer : "data scientist" ratio. Most of the talent is still from a physics background but there's more of an engineering bent where around half the team can probably write performance critical code when they need to. Elsewhere that ratio is much lower.

Determining which data to save is a mix of engineering, algorithms, physics, bits of machine learning, and (for better or worse) a bit of politics. Surprisingly we're always desperate for more talent there.

As you say, the goal is to try to stop problems before they affect the data, but it's not always perfect. Sometimes we discover sampling biases after the data comes in and need to correct for them, and in the worst case we sometimes blacklist blocks of data.

> It is a software engineer’s job to build quality software. A scientists job is to solve problems. There’s a clear boundary here, where the latter will deliver a concept to the former, who will eventually create a production-grade implementation off of that. Neither does a scientist have to build proper software, nor does a developer have to do cutting-edge research.

Precisely! See my relevant comment from another thread here - https://news.ycombinator.com/item?id=38821679

References:

1) Why science needs more research software engineers - https://www.nature.com/articles/d41586-022-01516-2

2) Research software engineering - https://en.wikipedia.org/wiki/Research_software_engineering

The real difference here is between hobbyists and mere workers. Programming happens to be one of the disciplines which has a lot of hobbyists. But that doesn’t mean you won’t find hobbyists in other disciplines.

Look at machining for example. Long dominated by people working in machine shops making tools and parts on the clock. But in the background there’s a strong hobbyist contingent and there you’ll find endless debates over whether a beginner should buy a decades old Bridgeport milling machine or a brand new Chinese-made one.

Similarly I find endless debates about what sort of frying pan to use (nonstick, stainless, carbon steel, or cast iron) among amateur chefs, but you’ll never see people working in a restaurant waste time on that.

In other words, it’s hobbyists who really obsess over tools and craftsmanship. They do it because they love it. Programmers just happen to be among the odd sort who can get paid to practice their hobby.

I don't know about that. Pros obsess over it too, but they don't bother to endlessly talk about it; they just do it. But if a better tool comes along, they'll consider it.

In computer science, for some reason - people struggle to distinguish between an engineer and a scientist.

I think that is, because we are still at the frontier and the lines between research and developing something new are quite blurry.

By now there are already lots of fields in IT that are quite standardized, but others not so much.

For example, what is the fastest way to draw lots of shapes on a canvas on the web?

There is no definite and fixed answer, as the field is still evolving and to find out the fastest way for your use case, you have to do research and experiments.

> as the field is still evolving and to find out the fastest way for your use case

No one in the world at large cares about the fastest way, they care about the lowest budget :)

Depends. If gaming is what you do, the better the performance, the bigger the market. As then more people can play your game.

HN is funny. I've been told on a previous discussion that AAA publishers don't optimize for potatos. Now you tell me it's a business requirement.

I told you it depends. There are AAA games for console and gaming computers and there are casual mobile games for example. Very different markets.

Oh the free to play stuff doesn't exist for me.

I guess the main problem is that many self-proclaimed "software engineers" are surprisingly bad "programmers" ;)

Just like most "research" is absolutely useless, when not counter-productive ;)

It's often very useful for those funding it.

Universities? Their main goal is to get better in the rankings, isn't it?

>It is a software engineer’s job to build quality software.

I think that most software engineers' jobs is not to build quality software, it is to make money. When you are trying to make money, the goal is not necessarily to make the best quality software that you can. Often, it is to make acceptably good software as soon as possible. A company that writes software that is half as good and ships it twice as fast might outcompete a company that writes software that is twice as good and ships it half as fast. I sometimes wish that my job was to build the best quality software that I can, but that is not the case. What I really get paid for is to make my employers' company successful, to make them money. And it's not that my employers don't care about making high quality software, it is just that if they cared about it too much they might get outcompeted by others who care less about it.

The problem is that most “professional” developers — i.e. people who write software as a career — are terrible at those things. Even (especially, sometimes) those that call themselves Software Engineers and talk endlessly about the right way to do things.

That is not to say the kind of software engineer that does what you say and tends to build quality software (or at least move it in that direction) doesn’t exist, but the demand for people who can make computers do stuff (loosely, developers) so far outstrips the supply that outside of a few bubbles (HN being one of them), they appear to be so vanishingly rare they might as well not exist.

I’m sure some scientists get to work with useful and highly valuable professional developers, but I’d be amazed if they were the majority.

If the bad wood working would jepordize the results of his, professional salary earning, work then he should probably consider learning wood working, no?

Don’t think so, no. A physicist has other stuff to learn and spend their time on. Instead, they should partner with a carpenter to do their woodworking from a rough sketch.

If your point is that the physicist should partner with someone who is a "professional programmer" ("carpenter") to do the coding, I couldn't disagree more, speaking as a former research physicist who wrote many programs while I was in academia. A "rough sketch" is not enough for a "carpenter" to go off of for the programs a physicist using computational techniques is writing. They'd need to have a sophisticated understanding of the physics and the mathematical model involved: which almost no "carpenters" have.

Now we’re lost in metaphors. A carpenter should definitely be able to build a pergola from a rough paper sketch, and a software engineer should be able to build a machine learning system from an algorithm paper by a data scientist.

That really depends on what you mean by "machine learning system." For a business, implementing some researched algorithm and essentially using a template? Sure, it's possible. The research that goes into producing the algorithm in the first place? Probably not.

There may be some classes of scientific programming which have simple enough models for a software engineer to implement. Scientific programming that relies on deep understanding of the domain and mathematical models employed to study it don't fit in that category.

Of course it always depends on the context. But my point still stands: a software developer is someone specialising in creating and maintaining code for an application. Any domain knowledge they may need for a particular application doesn’t matter here; their core skill set is software, whereas a scientists core skill set is active research.

Just because some scientific disciplines require extensive programming doesn’t make those scientists good software engineers: Producing an algorithm is different from generating revenue from it.

It’s like saying that a scientist doesn’t need to know how to write, and they should just pair up with ghostwriters/copywriters.

Many professions have tool/skill requirements that are not related to that profession on a strict sense, but are still necessary to do the job properly.

When I learned engineering we were taught how to draw diagrams by hand and write in a technical font. Computer code, for many science fields, is like diagrams but for theories.

It depends on what you consider a core skill. One has to specialize; that means leaving secondary tasks to others.

This is only true for systems where comparative advantage applies. Academia is not one of these places (as they explicitly don't want to be a part of the economy, which I understand).

> It’s like saying that a scientist doesn’t need to know how to write, and they should just pair up with ghostwriters/copywriters.

And I'd say that illustrates my point even better! A scientist needs to know how to write a paper, but that doesn’t make them a great author. When they aim to write a book, they should get help from a professional publisher. Both tasks require writing text, but a book is very different from a paper and requires heaps of additional training and knowledge that scientists usually don’t need to have.

> but that doesn’t make them a great author

This is also widely considered a huge issue in academia. The dissemination is so bad that a lot of great research is never read by others.

Academia is not a business. They barely can sustain themselves, therefore, they need to do things themselves. It is though to be a physicist. Mostly because you, besides being a great physicist, also need to be a good programmer and a good author.

Reply on news.ycombinator.com