dalke

I don't think it would be worthwhile to make your own hydraulic drill if there are rental alternatives.

More specifically, around 1981 my Dad rented a drilling rig to supply the irrigation pump for our yard in Miami.

See, the city was phasing out septic systems, and had installed sewers to our neighborhood. They charged for sewer based on water consumed. By drilling a well he could avoid paying sewage fees for irrigation water.

I think he hit water at 8 feet, and put the well down to 12. This was Miami.

It also drilled through some limestone.

My static pure HTML+CSS site is nearly 20 years old and renders just fine in modern browsers.

It's so old it has no https interface.

I do depend on my hosting provider to keep Apache and FreeBSD up to date, if only for security reasons.

Yes, I certainly expect so!

In my 2019 paper I earlier linked to I predicted "The VPOPCNTDQ instruction in the AVX-512 instruction set computes a 512-bit popcount, which should be faster still." because at the time I didn't have access to such hardware.

While those are easier to find now, I haven't revisited the issue because my experiments back then strongly suggested my code is now limited by memory bandwidth, not popcount evaluation performance.

I also don't know how many of my customers deploy on VPOPCNT-capable hardware.

My bitvectors have a 1-bt density of 5%, so I think the real next step is to look into something like BLOSC, where I store the data in compressed form, using a compression form which supports on-the-fly decompression faster than a memory read, then do the popcount on that transient data. 75% compression would need a 4x faster popcount.

I've tried to use BLOSC for this, but wasn't quickly able to figure how how to integrate it with my code, and realized it would likely require some pretty breaking big changes in my code that I couldn't really justify, so I've been putting it off for years.

I work in cheminformatics, and wrote one of the documents cited by Sagar.

The answer is "yes and no".

My area of focus is a part of molecular similarity. The overall idea is that molecules which are similar tend to have similar functionality. There are many approximate ways to estaimte molecular similarity. The one I focus on maps chemical substructures ("features" or "descriptors") to 1 or several bits on a bitstring of length ~1024 bits, called a fingerprint.

We use Jaccard similarity (called Tanimoto similarity in my field) of two fingerprints as a proxy for molecular similarity computed as popcount(A & B) / popcount(A | B). Since popcount(A) and popcount(B) can be pre-computed, this ends up being popcount(A & B) / (popcount(A) + popcount(B) - popcount(A & B). If the fingerprints are grouped by popcount then this boils down to computing popcount(A & B), plus some tiny amount of integer math.

This can be used to find the k-nearest matches to a single query, to cluster the fingerprints, to identify diverse fingerprints, and so on.

These methods bounded by two things: 1. the time needed to compute popcount of (A & B), and 2. the memory bandwidth.

The CPU bottleneck really is the popcount calculation. At https://jcheminf.biomedcentral.com/articles/10.1186/s13321-0... I compared different popcount implementations and found POPCNT about 2-3x faster than the fastest pure-C popcount.

However, POPCNT on Intel processors isn't all that fast. Rather, when I was really focused on this 5+ years ago, Intel processors only had one execution port than could handle POPCNT, so I could only get one 8 bytes per cycle. (Some AMD processors have several such ports, but I never tried one of those CPUs.)

Instead, Wojciech Mula, Nathan Kurz and Daniel Lemire showed that AVX2 instructions were even faster than sequential POPCNT instructions because AVX2 could handle more things in parallel. See my table for numbers.

For small bitstrings (I think 512 bits was the boundary) POPCNT is still the fastest.

With AVX2 it's fast enough that memory bandwidth becomes the limiting issue, and I need to start worrying about cache locality.

In case you are curious, here is the algorithm from that 1957 text book (2nd edition): https://archive.org/details/programsforelect00wilk/page/190/...

I never did sit down and figure out the notation.

It isn't in the 1951 (1st edition), at https://archive.org/details/preparationofpro0000maur

> Real maps show street names labeled on every street.

You are specifically describing street maps. There are other sorts of maps, even maps for areas with named streets where the streets are not named on the map.

A local orientation organization has checkpoints around many areas here in Sweden. You visit each one and tally up how many places you've visited. The web site lets you enter the 1- or 2-digit code for each checkpoint if you want to track it that way.

Here's my local map: https://koncept.orientering.se/provapaaktiviteter/hittaut/tr...

There are no street names on those streets. There are contour lines, making it a topographic map.

Look at https://en.wikipedia.org/wiki/Topographic_map and you'll see other examples of topographic maps showing streets but not listing all or even most of the street names.

I was cheating about the orange tree. That was the neighbors', whose tree came into our yard and we could get those oranges. They also had banana plants. My grandparents (across the street) had a huge grapefruit tree in their backyard.

I was happy when the guava tree blew down in a storm. Bugs got into the fruit so quickly, and I was the one who had to pick them up when I mowed. Yech! Mangos took longer to get to that state. And people would come into the yard to grab newly fallen mangoes, leaving less to deal with. A bit rude to do so without asking, we thought, but we couldn't them all anyway.

We had rented a car from San Francisco to San Diego. My wife was going to drop off the car at the airport. We asked at the hotel (by the harbor near Little Italy) how to get back. 'Uber'. We then looked at the map and realized it was about 0.5 miles away, with sidewalks the whole way. She walked.

My Houston experience was when I wanted to go to a nearby salsa place. It was about 3 blocks away from the motel, as the crow flies, but the other side of the interstate. It required a car to get there.

In geography class in college, the teacher talked about identifying old house locations sites in the Caribbean. I've forgotten the details over the last 30-odd years, but what they did was look for a place with multiple tree species with edible fruit. The idea was that if you found a small area containing, say, an orange tree, an avocado tree, a mango tree, a guava tree, and a lime tree there was probably someone living there.

(I picked those tree since they were the trees closest to my childhood house in Miami, not because I remember what the teacher said.)

Where did the 71 come from? The US inch was 2.54000508 cm and the UK inch was 2.5399977, until the Johansson's blocks convinced everyone to use the industrial inch of exactly 2.54 cm - what is now called the "international inch" (1ft = 0.3048 meters).

A nautical mile corresponds to 2.9229 cm. A survey foot is not much different than an international inch. I hope there's some oddity of history I'm missing!

Is that really true? My commercial airline pilot uncle didn't fly a glider until after he retired, but that's just anecdata.

I used to follow Patrick Smith, aka "Ask The Pilot". He didn't start with a glider.

Now I follow Scott Manley. He just got his private pilot license, and did not start with a glider. Nor did my missionary pilot uncle. Nor did a co-worker of mine who was learning to fly.

Hereabouts I see gliders sometimes, but I far more often see powered aircraft. I get the impression there are far more powered flight schools than glider schools; https://www.flightschoollist.com/california-glider-schools/ lists 13 glider schools in California while https://www.flightschoollist.com/california-airplane-flight-... list 91 for powered flight.

The chlorine level isn't the issue - it's the urea reacting to the chlorine.

There was even some neat research a few years back to figure out how much came from urea from pee, vs. from other bodily secretions: https://www.npr.org/sections/health-shots/2017/03/01/5177859... .

> The scientists calculated that one 220,000-gallon, commercial-size swimming pool contained almost 20 gallons of urine

After reading that NPR article, I can suggest another possible alternative; the pools might drain and refill their pools more often, rather than top-off when it gets low.

Dry hair might not be that good of an indicator? I mean, I see people doing breast stroke with their head completely out of the water.

Swedish pools at the time required you to shower, naked [1], before entering the pool, and to use soap while showering, with special attention to the hair, armpits, crotch, and feet. This helps get rid of the dirt and urea (which when combined with chlorine makes the 'swimming pool smell'), keeping the pool clean and with little odor.

It is difficult to wash underwear with the same thoroughness. I have not seen people laundering their underwear before swimming.

[1] Our local pool changed to allow showering in one's bathing trunks. I personally don't like the rule, as I think people will skimp on washing their crotch before entering the pool.

I'll forward your thanks to my Mom!

Years ago I ported some software from IRIX to some Linux-based OS. There were a few differences in the result. I eventually tracked it down to how on IRIX char is unsigned while the Linux-based OS used signed char.

The program used a 'char' to store the value of the formal atomic charge of an atom, which is typically 0, but for the sorts of chemistry I deal with can be +1, +2, -1, or -2, so it makes sense to allocate only 8 bits to store the 'char'ge. :)

On IRIX, where they had deployed the code for years, the charges were actually being interpreted as 0, +1, +2, 255, and 254.

Going back to the code example you gave, isn't there also an issue if read failure occurs on the first byte? Then `out` is 0, returning -0, which is 0, giving no way for the caller to distinguish between EOF and read failure.

Never did, but going back to "exotic", when I moved to Illinois I encountered a lot of new vegetation. "What's this one?" I asked. My friends looked at me with an odd expression. "Maple." Me: "Oh! It does look like the Canadian flag!"

To them, mango, avocado, and Key Lime trees - all which we had in our yard - were exotic. To say nothing of lizards and iguanas, nor having a flock of parakeets fly out of a tree.

As a little kid growing up in Miami, I had to learn that jai alai wasn't well-known elsewhere in the US. I remember watching the ads for it on TV, though (like dog racing) I was too young to go.

For me, I thought the large school lockers I saw on TV were pretty exotic. Why were they large enough that someone could hide in them? Ours were about backpack sized.

My Mom explained that's where people put their winter clothes, which also explained why the lockers were always inside enclosed hallways. I was used to having hallways with one side open, to allow a breeze.

To show you what I mean, here's my elementary school, https://www.google.com/maps/@25.7538402,-80.2581714,3a,27.4y... . They didn't install A/C until I was in 4th grade. The entire classroom wall facing the hallway was a bank of doors that could open up to allow a cross-breeze, with a wall of windows on the other side.

For that matter, snow was pretty exotic too. ;) When I went up north to Tallahassee for college, us S. Florida people were so excited when the temperature got to freezing. We stayed up late to see if the water we left out would freeze.

What git links? I use Mercurial.

Oh, I see. Some of the commit links result in "502 Bad Gateway" from nginx. How odd!

Even stranger, I can't reproduce it on my other repos.

I so rarely use the web interface I never noticed. I mostly use it via the command-line, for solo projects, effectively as off-site backup.

EDIT: Yeah, looks like that issue was reported 10 days ago, at https://todo.sr.ht/~sircmpwn/hg.sr.ht/46 , with no response from the operators.

As a Mercurial user and fellow traveller to free software, it's hard to find a good and responsive hosting provider.

I thought the README was long enough that I wasn't going to go into those details, especially as 1) I am not an expert on the topic, 2) the choice of curve depends on your application, and 3) there are a lot of resources on the topic. I listed a few.

If you want to learn more, see "Space-Filling Curves An Introduction with Applications in Scientific Computing", listed as one of my sources, eg, "Algorithm 13.3: Matrix-vector multiplication based on Hilbert order matrix traversal" and "Matrix Multiplication Using Peano Curves".

One of the sources I cited is "Improved Data Locality Using Morton-order Curve on the Example of LU Decomposition" at https://ieeexplore.ieee.org/document/9378385 :

> Here, we look at the block factorization algorithm, where the LU decomposition performance depends on the performance of the matrix multiplication. In both cases, the LU decomposition and the matrix multiplication, such a matrix is traversed by three nested loops. In this paper, we propose to traverse such loops in an order defined by a space-filling curve. This traversal dramatically improves data locality and offers effective exploitation of the memory hierarchy. Besides the canonical (or line-by-line) access pattern, we demonstrate the traversal in Hilbert-, Peano and Morton order. Our extensive experiments show that the Morton order (or Z -order) and the inverse Morton order (or И-order) have a better runtime performance compared to the others.

I first learned about this issue from "The Anatomy of High-Performance 2D Similarity Calculations" at https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4839782/ :

> Certain self-similar space-filling curves provide a natural hierarchy and iteration order to optimize locality in a cache-oblivious algorithm. In particular, Morton ordering, based on the hierarchical Z-order curve, has been previously used to optimize cache locality in matrix multiplication,22 suggesting that it may also be useful for similarity matrices. .... Other space-filling curves, such as the Hilbert curve, also have excellent cache-locality properties; however, the calculations required to implement the Z-order curve are particularly simple.

I want to reproduce their result, then see how other traversal orders affect the timings. Careful reading of that paper shows they only used sizes which are a power of two, while real-world data sets aren't so convenient.

I couldn't find an off-the-shelf library with non-powers-of-two/rectangular generalizations, so I wrote one.

It's definitely not showing the effect of memory fetch prediction, as the entire benchmark should fit in L1 cache.

The point of the benchmark was to show how much overhead there is for using the method. It doesn't do any memory traversal at all - it's the iterator equivalent of:

  long long benchmark_for_loop(int w, int h, int bad_i, int bad_j) {
    long long count = 0;
    for (int i=0; i<w; i++) {
      for (int j=0; j<h; j++) {
        if (i == bad_i || j == bad_j) {
          return INVALID;
        }
        count++;
      }
    }
    return count;
  }

What the zigzag benchmark shows is the overhead for the few additional operations needed to figure out if it's traversing left or right; more specifically the "if (row % 2 == 0)" test and the else clause of the following:

    *i = row = local.pos / local.h;
    col = local.pos % local.h;
    if (row % 2 == 0) {
      *j = col;
    } else {
      *j = local.h - col - 1;
    }

The factor of 4 performance difference between the for-loop and the row iterator is all due to function call overhead for the iterator API.

Just posted my code as a "Show HN" at https://news.ycombinator.com/item?id=36762283 . The actual project is at https://hg.sr.ht/~dalke/rectfillcurve . I've implemented a few space-filling curves and wrapped them into an iterator API.

That did the trick - thanks! I couldn't figure out what the [] meant in this context and thought it was to make a recursive call.

(I've been working on basic space-filling curves, and came across that Wikipedia page last week, which is about all I know about L-systems.)

I can't figure out the grammar. I wanted to do the Hilbert curve at https://en.wikipedia.org/wiki/Hilbert_curve:

    Alphabet : A, B
    Constants : F + −
    Axiom : A
    Production rules:
        A → +BF−AFA−FB+
        B → −AF+BFB+FA−

I thought it might be:

  A=|+[B]|−[A]|[A]-|[B]+
  B=|−[A]|+[B]|[B]+|[A]−

but that doesn't work, and I don't know what's wrong.

Along the same lines, what about using the first `n` outputs from a random number generator? (With "it depends on the PRNG" replacing your "it depends on the hash function".)

I come from cheminformatics, where we've used a form of superimposed coding very similar to a Bloom filter. In one incarnation, developed in the 1980s, enumerate all linear structures of up to length 7, determine a unique order based on atom and bond characteristics, convert that into a PRNG seed, and generate (typically) 4 values, each used to set a bit in a bitstring of size ~1024 bits.

I write similar because (1) in some cases the number of bits is not fixed, eg, it may vary depending on the structure, and (2) it uses a PRNG instead of `n` different independent hash functions.

I'm therefore curious if PRNGs have been examined in a Bloom filter context, or for MinHash, which also has `n` different hash functions. Then I can drop (2) from my list of differences.

That's a construction issue.

If you're used to the stick-built homes and ballon framing, then the article quote from Vanwyck is right: “The majority of North Americans haven’t experienced high-quality housing, and so they also don’t know what they’re missing and what they can ask for.”

In the apartment house I live in Sweden, I hear less of my neighbors than when I lived in a detached house in the US. The walls and floors here are SOLID! I had to get a impact drill so I could install curtains, the concrete is that tough.

Some apartment complexes have a private courtyard garden where you can bbq and play with the kids (and play with the dogs, though you'll need to go elsewhere to let them pee and poop), and have the upkeep paid by your rent.

Ours doesn't, but our nearest park is across the street and the nearest big playground 5 blocks away, see https://www.google.com/maps/@58.2841034,12.2930193,3a,75y,23... . And we have several large shared rooms in the basement where the kids can play on wet dreary days, or hold a birthday party or, in my case, work during the COVID restriction years.

Now, I hate mowing and gardening, which is why I also paid someone to do it for me in the US. For those who like it, I can totally see how having a yard would support that hobby.

It's just odd to see code I worked on in a very small way back in the 1990s - the molecular dynamics program NAMD - be used in AMD marketing literature.

At college in Tallahassee my friends would have a stronger accent when talking on the phone to family than talking to friends, even at a Southern college.

This is either part of, or related to, "code-switching". https://en.wikipedia.org/wiki/Code-switching . That could be the case at your FANG company, for one or multiple of the reasons given.

Well, speak of the devil and he doth appear! :)

I looked at PEP 708. I was confused by what "repository" means. In PEP 503 "A repository that implements the simple API is defined by its base URL .... Within a repository, the root URL (/ for this PEP which represents the base URL) MUST be a valid HTML5 page with a single anchor element per project in the repository.".

A repository contains projects - "individual project contained within a repository".

PEP 708 seems to use "repository" to mean both that and individual project. Consider "To enable one repository to extend another, this PEP allows the extending repository to declare that it “tracks” another repository by adding the URL of the repository that it is extending."

The examples show project with new entries tracking a project on another repository.

This made it hard for me to understand what something like "repository owner" really means.

> If I understand your uses well enough you should be able to add a line or two of HTML to your repository and not have any interruptions or warnings.

I'm not sure it works, at least, not fully.

I tell people to use:

  python -m pip install chemfp -i https://chemfp.com/packages/

This used to contain only one project, "chemfp".

Now it also contains "click" and "tqdm" entries, copied verbatim from the respective PyPI project entries, because I recently added my first required install dependencies, and -i doesn't automatically fall back to PyPI.

I use '-i' because I don't want installs to start using the old chemfp version on PyPI. (Why? I only distribute pre-compiled wheels for 'manylinux'. I didn't release Python 3.11 wheels until a few weeks ago. I don't want pip for 3.11 users, or on macOS, to find the source version and try to install them. And "We’ve spent 15+ years educating users that the ordering of repositories being specified is not meaningful, and they effectively have an undefined order." ;)

I would prefer to not maintain copies of the click and tqdm project entries, as I need to remember to refresh them.

I think with PEP 708 I can have a single

    <meta name="pypi:alternate-locations" content="https://pypi.org/simple/click/">
    ...
    <body></body>

which will fix that pain point.

However, the main issue I have is namesquatting is still too easy. If I started chemfp now, with no PyPI entry, then from my side nothing changes.

But I've had people do "pip install chemfp" WITHOUT the -i option then ask what why it didn't work.

I assume that's because people aren't used to using a -i (or configuring it in their requirements.txt) so aren't sensitive as to why it's important.

Namesquatting a purely non-PyPI project then comes easy - register it on PyPI. PyPI is active about namesquatting, but you all surely don't track all small non-PyPI projects.

What I would like is, I thought, pretty simple:

    python -m pip install chemfp.com:/package/chemfp/

with the ability to also specify a path like that in the requirements.

That's considered in PEP 708 as possible ("To my knowledge the only systems that have managed to do this end up piggybacking off of the domain system and refer to packages by URLs with domains etc") but rejected ("our ability to retrofit that into our decades old system is practically zero without burning it all to the ground and starting over" ... "This would upend so many core assumptions ...")

This means while some of my current issues will be assuaged with this PEP, my fundamental concern will not.

I also don't have a smart phone or hardware key, and feel the same.

I host my packages on my own server using the PEP 503 "Simple Repository API". (For what it's worth, Donald Stufft, author of this PyPI blog post, also wrote PEP 503.)

I migrated away from PyPI years ago because modern FOSS distribution is increasingly intermediated by package managers, which I think worsens the connection between user and producer by adding a layer of anonymity.

I see 2FA as a technical work-around using mechanical trust instead of personal trust.

I don't care if people who don't know about me or my project web site can't find it on PyPI, because care is reciprocal - I want them to care about me too, at least enough to know where the project came from. That's where I think trust comes from.

My packages will never be popular. I develop very specialized software for a field with perhaps 10,000 world-wide users. No one will pay for my software, or hire my services, if they only visit PyPI.

I thankfully still have a Python 2.7 version of my software on PyPI. I haven't deleted it, to make name-squatting harder. It's even lead people to contact me directly, to ask about Python 3 support. I never would have heard from them, had I published the newest versions on PyPI, and that direct contact breaks the anonymity PyPI interposes.

A nice plus is how hosting means I have full access to the download logs, giving me insight onto what platforms people use that PyPI doesn't provide, because "anonymity".

The biggest annoyance is they main reason I started using PyPI was because someone else added another project of mine to PyPI.

I distributed https://pypi.org/project/PyRSS2Gen/ on my own web site as source. Someone else added it to PyPI, with me listed as the primary developer. I didn't know about it until the PyPI devs emailed developers like me to encourage them to support Python 3! I then had to ask that stranger for access to the PyPI entry.

I fear that will happen again, if I ever develop a new package that I distribute myself.

I hope this provide something for you to chew on, as you consider what you might do.

I did basically pull a rabbit out of the hat. No DM would let me get away with that, but those events will be uncommon in the training corpus.