Hacker News Reader

I'm one of those scientists, working in industry R&D. I use Python for everything from Jupyter notebooks to a roughly 8000 line program that runs a prototype of a product. Python pulled me away from Visual Basic several years ago, and then I have a longer history going back through Turbo Pascal and ultimately originating with primordial BASIC. On the hardware side, I use C within the Arduino ecosystem. I've got roughly a half dozen other scientists at my work site using Python at various levels.

Confession: Code quality is a problem. I care a lot about code quality. My team recently brought on a couple of people with real programming background, and I've asked them to help me bring myself up to date on good techniques, and it has helped. I was lucky that some of my former training, e.g., in Pascal, was not just about how to program, but actually how to write good code (for its time).

But here's my view. Don't just tell us that our code sucks. We already know it, but we're under the same pressures as anybody else, and programming is a force multiplier that we won't give up. We need instructions that we can understand at our level, that will help us do this stuff better, perhaps incrementally. Starting with a strict discipline is hard, but Python supports the ability to go back through working code and improve it.

I'm working with some researches on about ~40k lines of Python disease modelling code and these are the things that helped us becomes more productive:

- tests (unit, smoke, acceptance) (eg. https://mattsegal.dev/alternate-test-styles.html, https://understandlegacycode.com/approval-tests/)

- continuous integration (via GitHub actions eg: https://mattsegal.dev/pytest-on-github-actions.html)

- 10x performance improvements via cProfile (https://julien.danjou.info/guide-to-python-profiling-cprofil...)

- getting fast feedback on code changes by visualizing outputs with Streamlit (https://www.streamlit.io/)

- input parameter validation (eg. https://pydantic-docs.helpmanual.io/, https://mattsegal.dev/cerberus-config-validation.html)

- total automation of data processing job (eg. https://www.prefect.io/ - we don't use this but that's the kind of thing we have)

Your post reminds me of a segment[0] of a Pycon keynote where Jacob Kaplan Moss talks about "real programmers". One of the anecdotes was that he was hiring programmers and he stumbled on a student who had worked on a project who told him she can't apply because she "isn't really a programmer".

>I've asked them to help me bring myself up to date on good techniques

Here's an old draft of our onboarding document[1], and a few comments here on HN about this[2].

PS: in case anyone is wondering how I found the segment in question, I use a Chrome extension called "YouTube Captions Search"[3] went to the video, activated closed captions, clicked on the extension's icon, and searched for "geographic" because I remembered that the system the student had built was a geographic information system (GIS).

- [0]: https://youtu.be/iOp0mSDCh6c?t=1084

- [1]: https://jhadjar.gitlab.io/kbase/hiring/#resources-to-learn-o...

- [2]: https://news.ycombinator.com/item?id=15576496

- [3]: https://chrome.google.com/webstore/detail/youtube-captions-s...

This is an area I want to get more into actually. I've done a bunch of freelancing for scientists, and it seems to me that there is now a wave of scientists hitting the code size and complexity level where understanding the principles of maintainable software architecture is become pretty important. One of the gigs was reverse-documenting some RCPP because they had been left with a ball-of-mud someone had to write in a hurry and no one knew how to break it up. It was interesting, and cool to help people doing real science. If you have any thoughts about the viability of teaching architecture to scientists, I'm all ears. You can find me on linked on or my user name plus the email of a behemoth starting with G... ;-)

As to how they get the numbers, this is TIOBE's methodology: [1]

   Since there are many questions about the way the TIOBE index
   is assembled, a special page is devoted to its
   definition. Basically the calculation comes down to counting
   hits for the search query
   
   +"<language> programming"

   In the next few sections it is explained what search engines
   qualify, what programming languages qualify and how the
   ratings are exactly calculated.
   
   Search Engines

   There are 25 search engines that are used to calculate the
   TIOBE index. The selected search engines are the 25 highest
   ranked websites of Alexa that meet the following conditions:
   
   The entry page of the site contains a search facility
   The result of querying the site contains an indication of the number of page hits
   The results should be available in HTML with clear tags
   Search engines in languages with special characters should be encoded properly
   The search engine should at least return 1 hit for 1 query
   The results of querying the site shouldn't contain too many outliers
   Porn sites are excluded

[1] https://www.tiobe.com/tiobe-index/programming-languages-defi...

> Chaturbate.com: PORN_SITE

Gave me a laugh. Engineers would initially qualify a porn search engine to search for this sort of experiment

That'd certainly give "Python" a leg up as a search term.

Well, yes. Why wouldn't you? They hire engineers, and they have programming projects. Meets all requirements.

I don't usually search pornhub for programming help

From what I've seen, jupyter notebooks reach convoluted excel formulas-level of accessibility and efficacy for lay people.

Most of us really underestimate what "non" programmers can achieve with if/else, loops, equality tests, a repl and autocompletion.

Yeah I'm guessing code quality is _tanking_ around the world. Lots more code , a lot less quality.

Perhaps, but when I built the shed in my back yard I didn't enforce the same standards a construction engineer would for a skyscraper. I increased the quantity of "shitty construction" in the world, but it works great for my purposes.

Not every programmer is a "software engineer" for many, code is a tool to get a job done, it's a means to an end, and that's wonderful.

And a lot more stuff being done. Cut it out with the elitism.

It's like being an anaerobic bacteria complaining about the abundance of oxygen in during the Cambrian explosion.

"Back in my day no stinking, puny, oxygen breathing things were around, only sturdy creatures such as myself" :-)

Yeah, it was messy, but that's how we were able to have humans, hundreds of millions of years later.

What is "code quality" anyway?

Just want to bump this from another perspective: I’m an engineer in a big EU science infrastructure project and everything has a Python API. The main lab environment is JupyterLab. No one wonders whether what languages scripts or workflows will be written in.

Julia is on the radar but even in Julians’ wildest dreams Python remains relevant through PyCall. Why rewrite it if you can reuse it? (Obv I refer to boring-to-write non-performance-sensitive code here)

Ya, I think it's great that Python has become a sorts of universal / unified language these days. When I was a student, people used a bunch of different languages to write software, depending on what group they were in, or what the culture there was.

You had Matlab, R, Octave, Java, C++, Lisp, etc. all depending on what type of scientist or engineers you worked with (Statisticians used R/SPSS/etc., Physicists and Electrical Engineers used Matlab or C++, Computer Scientists used Lisp etc., and the list goes on)

At least today, most is written in Python - only the libraries are different.

Also, sharing code is a breeze today. Back then you'd have to download or manually transfer the codebase from other users, check dependencies, install software, check licensing, and what not.

Today you can fire up a Python notebook, and that's it - very nice for smaller stuff. Even larger applications is a easy as pie.

>Physicists and Electrical Engineers used Matlab or C++

Trained as an EE and this is true. We used MATLAB/Simulink for signal processing and control systems. Octave, too (because our sensor technology teacher was more comfortable with that). x86 and Microship Assembly, C, and Pascal. Used LTSpice and NI Multisim for circuit design and simulation. Cadsoft EAGLE for printed circuit board (PCB) work. Played a bit with LabVIEW.

However, I started coding as a child with BASIC/x86 Assembly/C/Pascal/VB/Delphi. Doing mostly Python for the past several years. I suppose one could find reasons to "love" or "hate" anything, but I always found that nuance keeps sanity.

> I think one thing programmers underestimate is how many scientists and academics are using Python. It's not just ML, it's pretty much every kind of scientist and tons of non-science academics too. The fact that it's a great "casual" or part-time language is huge for these people. They aren't full time (or even self-identifying) programmers, but their programs are non-trivial. The readability, portability, tooling, and massive package ecosystem is huge for them.

I've found the reliability and tooling really unsatisfactory, at least compared to Haskell.

The libraries are there, though, and familiarity is in the departments (psychology (maybe more R?), physics, &c.)

Oh yeah. Lots of scientists just want to turn their text file of data points into a graph, and Python is great for that.

Some people just stick to the basic stuff but there's plenty of other, more complicated programs that are developed by academics/students. Guess what they're going to use? The language they first used to make that pretty graph.