Comment by madaxe_again | Hacker News Reader

Comment by madaxe_again | Hacker News Reader

> It may have gotten a boost from using Python, and not Maple (Lance’s favorite program for math) or Mathematica (mine), and it may have used much better software engineering practices than we would have, but not super-intelligently so.

This is the crux of the matter here. Physicists are physicists. They are not software engineers. I read physics between 2001 and 2005, and the programming language they had us use for all of our assignments was FORTRAN 77. They were still trying to decide if it was ok to move students on to FORTRAN 95.

FORTRAN is pretty performant - if you know what you’re doing. Very, very few people did - and they, me, went on to have careers in software, not physics. I remember some FITS (astronomical image format) processing software someone was using to calculate ephemera - and it took DAYS to run over a few thousand images and produce an output. I sat down with it, screwed around for an afternoon, and it produced a result in under a minute - so much faster that I honestly thought I had broken it - but I hadn’t. It was just terribly written by someone who was excellent in their domain and terrible at writing code.

I see there being an enormous opportunity here, in AI providing scientists with software that isn’t diabolical.

Exactly the same case here. A few years ago, pre-ChatGPT, my wife was working with a massive database in SPSS or R (can't remember which). Now these "databases" are really just huge CSV/TSV files, with millions of records, one record per line. She was applying some very complicated mathematical functions to every single row, and she was frustrated because it was going to take days to apply a particular calculation she was working on to the whole database.

So I took a look for about 30 mins figuring out the code (I don't know SPSS or R), and it turns out she was doing some calculations that were identical for every single line, regardless of that line's fields. So I pre-calculated a few things, prior to starting the per-record-loop, and then just used the pre-calculated values every time. After that, every run took less than 5 minutes. She's amazing at math and economics, but optimizing low-level code is not something she has done in the past.

Yeah, very similar. For each and every image, the code loaded every image into memory, several times each, then did a pixel-by-pixel walk through the entire image, identified all objects, and then compared all of the objects to all of the objects from all of the previous images (which it would not use the calculated result for, but rather recompute that too) to establish which one was moving - or which one was staying still.

The main fix was not making it hideously IO bound, as not only was it loading every image multiple times, it was then grinding away in swap as we’re talking tens of gigabytes of raw image data, in an era when 1gb of ram was a lot - and then making it spiral out from the last known location of the object of interest rather than brute-forcing it. My solution wasn’t even optimal, as it was literally just an afternoon of tooling around as a favour.

Reply on news.ycombinator.com