aktau

Glad you like it. Hope you get as much use of it as me.

The default storage location for memo(1) output is /tmp/memo/${USER}. Most distributions either have some automatic periodic cleanup, and/or wipe it on restart.

Separately from that:

  - The invocation contains *memo* right in there, so you (the user) knows that it might memoize.
  - One uses memo(1) for commands that are generally slow. Rerunning your command that has a slow part and having it return in a millisecond while you weren't expecting it should make the spider-sense tingle.

In practice, this has never been a problem for me, and I've used this hacked together command for years.

Another thing I built into memo(1) which I forgot to mention: automatic compression. memo(1) will use available (de)compressors (in order of preference: zstd, lz4, xz, gzip) to (de)compress stored contents. It's surprising how much disk space and IOPS can be saved this way due to redundancy.

I currently only have two memoized commands:

  $ for f in /tmp/memo/aktau/* ; do 
      ls -lh "$f" =(zstd -d < $f) 
    done
  -rw-r----- 1 aktau aktau  33K /tmp/memo/aktau/0742a9d8a34c37c0b5659f7a876833b6dad9ec689f8f5c6065d05f8a27d993c7bbcbfdc3a7337c3dba17886d6f6002e95a434e4629.zst
  -rw------- 1 aktau aktau 335K /tmp/zshSQRwR9

  -rw-r----- 1 aktau aktau  827 /tmp/memo/aktau/8373b3af893222f928447acd410779182882087c6f4e7a19605f5308174f523f8b3feecbc14e1295447f45b49d3f06da5da7e8d7a6.zst
  -rw------- 1 aktau aktau 7.4K /tmp/zshlpMMdo

That's roughly 10x compression ratio.

I have a bunch, but one that I rarely see mentioned but use all the time is memo(1) (https://github.com/aktau/dotfiles/blob/master/bin/memo).

It memoizes the command passed to it.

  $ memo curl https://some-expensive.com/api/call | jq . | awk '...'

Manually clearing it (for example if I know the underlying data has changed:

  $ memo -c curl https://some-expensive.com/api/call

In-pipeline memoization (includes the input in the hash of the lookup):

  $ cat input.txt | memo -s expensive-processor | awk '...'

This allows me to rapidly iterate on shell pipelines. The main goal is to minimize my development latency, but it also has positive effects on dependencies (avoiding redundant RPC calls). The classic way of doing this is storing something in temporary files:

  $ curl https://some-expensive.com/api/call > tmpfile
  $ cat tmpfile | jq . | awk '...'

But I find this awkward, and makes it harder than necessary to experiment with the expensive command itself.

  $ memo curl https://some-expensive.com/api/call | jq . | awk '...'
  $ memo curl --data "param1=value1" https://some-expennsive.com/api/call | jq . | awk '...'

Both of those will run curl once.

NOTE: Currently environment variables are not taken into account when hashing.

But there is a grain of truth to their commentary. "Modern" gets old fast, it probably shouldn't be used, just like "new" shouldn't be used in project/library names.

IMHO, removing "modern, flexible and scalable" would improve that line.

I've diagonally read through the README, and for my taste there is overuse of "soft" adjectives (simple, easy, ...) which makes me (an engineer) discard it as marketing-driven.

Thanks for that reference. Do you know if the JVM still uses thin locks? Did they migrate to thin locks? I ask because I found a 9 year old reference with a JVM calling futex: https://stackoverflow.com/questions/32262946/java-periodical....

> In theory compare-and-swap or the equivalent instruction pair load-exclusive/store-conditional are more universal, but in practice they should be avoided whenever high contention is expected. The high performance algorithms for accessing shared resources are all based on using only fetch-and-add, atomic exchange, atomic bit operations and load-acquire/store-release instructions.

> This fact has forced ... there were no atomic read-modify-write operations, so they have added all such operations in the first revision of the ISA, i.e. Armv8.1-A.

I'm not sure if you meant for these two paragraphs to be related, but asking too make sure:

  - Isn't compare-and-swap (CMPXCHG on x86) also read-modify-write, which in the first quoted paragraph you mention is slow?
  - I think I've benchmarked LOCK CMPXCHG vs LOCK OR before, with various configurations of reading/writing threads. I was almost sure it was going to be an optimization, and it ended up being inobservable. IIRC, some StackOverflow posts lead me to the notion that LOCK OR still needs to acquire ownership of the target address in memory (RMW). Do you have any more insights? Cases where LOCK OR is better? Or should I have used a different instruction to set a single bit atomically?

Reminds me a bit of Steve Yegge's latest [^1]. He gives an LLM full control over his editor (Emacs) by allowing it to call eval (as I understand it). He doesn't talk about which guardrails (if any) he put on this.

[^1]: https://x.com/Steve_Yegge/status/1942336357650817235

Not sure if there is already quorum on what a solution for adding labels to non-point-in-time[^1] profiles like the heap profile without leaking looks like: https://go.dev/issue/23458.

[^1]: As opposed to profile that collect data only when activated, like the CPU profile. The heap profile is active from the beginning if `MemProfileRate` is set.

This goes straight into my reference list. Sandboxing a process is confusing on Linux.

I appreciate that the article focuses on approaches that drop privileges without having root oneself. I've seen landlock referenced at time (https://lwn.net/Articles/859908/), but never so clearly illustrated (the verbosity feels like Vulkan).

Out of curiosity, I'd wish even more approaches were compared, even if they require root. I was about to mention seccomp-bpf as an approach that requires root, but skimming the LWN article I posted above I find: "Like seccomp(), Landlock is an unprivileged sandboxing mechanism; it allows a process to confine itself". It seems like I was wrong, and seccomp could be compared/contrasted.

This is one area where Gerrit Code Review is (was? I don't know if it changed) is superior. It stores everything it knows about in git repositories (preferences in a separate meta git repository, comments, patches). With the right refspec, you can pull it all down and have a full backup.

Does anyone know how this contrasts with `text-align: justify` (mentioned in https://news.ycombinator.com/item?id=43258709#43260606)?

In Chrome (can't test Safari), `text-wrap: pretty` has a much milder effect.

Should one use both together in the main text of your average blog? I checked, they do appear to make individual changes.

I have the opposite experience. I find Cortazar impenetrable, but the Borges stories "speak to me".

I should correct and clarify: I meant 3-4x more expensive in relative terms. Meaning:

  - For C++ programs, the allocator (allocating+freeing) consumes roughly 5%  of cycles.
  - For Go programs, the allocator (runtime.mallocgc) used to consume ~20% of cycles (this is the data I referenced). I checked and recently it's become closer to 15%, thanks to optimizations.

I have not tested the performance differential on a per-byte level (though that will also differ with object structure in Go).

It is not. Please see my answer (https://news.ycombinator.com/item?id=43545500).

Side note: see https://tip.golang.org/doc/gc-guide for more on how the Go GC works and what triggers it.

GC frequency is directly driven by allocation rate (in terms of bytes) and live heap size. Some examples:

  - If you halve the allocation rate, you halve the GC frequency.
  - If you double the live heap size, you halve the GC frequency (barring changes away from the default `GOGC=100`).

> ...but if you look at pprof of a Go app, GC mark phase is what takes time, not GC sweep.

It is true that sweeping is a lot cheaper than marking, which makes your next statement:

> Short lived allocations, those which GC mark phase will never reach, has almost neglible effect on GC times.

...technically correct. Usually, this is the best kind of correct, but it omits two important considerations:

  - If you generate a ton of short-lived allocations instead of keeping them around, the GC will trigger more frequently.
  - If you reduce the live heap size (by not keeping anything around), the GC will trigger more frequently.

So now you have cheaper GC cycles, but many more of them. On top of that, you have vastly increased allocation costs.

It is not a priori clear to me this is a win. In my experience, it isn't.

Comparing with a fairly optimized malloc at $COMPANY, the Go allocator is (both in terms of relative cycles and fraction of cycles of all Go programs) significantly more expensive than the C/C++ counterpart (3-4x IIRC). For one, it has to do more work, like setting up GC metadata, and zeroing.

There have recently been some optimizations to `runtime.mallocgc`, which may have decrease that 3-4x estimate a bit.

Upstream thinks a type-safer `sync.Pool` is a good idea too. It's being discussed in https://go.dev/issue/71076.

I've experienced quasi-remission twice now. Both times when I got so sick from food in a foreign country that I couldn't eat for days (and had no appetite for it either). I lived on water. Afterwards, for 1-2 weeks, I did not require insulin (while eating a lot).

I used to think it was due to the pills they gave me there, or perhaps due to the bacteria (or virus?) causing some strange temporal abating of auto-immune response and regrowth of beta-cells. But seeing this is making me reconsider that (I was doubting the effect of the medicine since I took some home and took it in a healthy state but did not get the good effects).

That sounds like double-buffering. Or did I misunderstand? Is it actually a ring-buffer? Can you give more details? What is a Half-Transfer interrupt?

I actually like `nvim --clean`, which is just the basics. Early on in the Neovim project, a lot of heirloom defaults were changed to be more modern, resulting in a better (IMHO) out-of-the-box experience. I use `nvim --clean` as my man-page viewer:

  MANPAGER=nvim --clean -c "colo sorbet" +Man!

Startup speed is blistering.

My current config is pretty stable, and not that large. But if it were causing issues, I'd seriously consider only doing LSP setup, which is not that onerous with the latest APIs (it was already fairly easy with `vim.lsp.start`, but `vim.lsp.config` and `vim.lsp.enable` make it easier still: https://neovim.io/doc/user/lsp.html).

I highly recommend this to anyone reading HN. I wouldn't say it changed my life, but I think about Borges and Fictions/The Aleph at least once per year.

Copying makes for surprising semantics, and prevents some representation changes.

An example w.r.t. the surprising semantics:

  var ms []mypb.Message = ppb.GetMessages() // A repeated submessage field
  for i, m := range ms {
    m.SetMyInt(i)
  }
  assert(ppb.GetMessages()[1].GetMyInt(1) == 1) // This would fail in general, due to SetMyInt acting on a copy.

This would not work as expected, as I highlighted in the comment. Basically, acting on value types means being very careful about identity. It makes it easy to make mistakes. I like data-driven code, but working around this (sometimes you'd want a copy, sometimes you wouldn't) would be a painful excercise.

You may have noticed that changing a heavily pointerized tree of types into value types often compiles with just a few changes, because Go automatically dereferences when needed. But it often won't work from a semantic point of view because the most intuitive way to modify such types uses copies (the range loop is a good example).

Now imagine changing the representation such that it carries a mutex, or another nocopy type. That would lead to issues unless those nocopy types would be encapsulated in a pointer. But then you get issues with initialization:

  var m mypb.Message // Value type, but what about the *sync.Mutex contained deep within?

Also consider laziness

  func process(lm mypb.LazyMessage) {
    if lm.GetSubMessage().GetInt() != 42 {
      panic("user is not enlightened")
    }
  }

  var lm mypb.LazyMessage
  process(lm) // Copy a fairly large struct.
  ln.GetSubMessage().GetInt() // Does lazy unmarshaling of sub_message redundantly.
  

If you want to make the argument that individual messages should be pointers, but slices should still be value slices. Then I have the following for you:

  ms := m.GetSubMessages() // []mypb.Message
  el := &ms[0]
  anotherEl := new(mypb.Message)
  ms.SetSubMessages(append(ms.GetSubMessages(), anotherEl)) // Can cause reallocation, now el no longer references to m.GetSubMessages()[0]. But it no reallocation happened, it does.

In practice, value typing leads to a bunch of issues.

Since you seem so sure of your position, I'm actually curious. How would you design the API, how would you use it? Do you have any examples I can look at of this style being used in practice?

I would.

> > - Messages (especially opaque ones) are not supposed to be copied.

> So?

If you have a []mypb.Message, and range over it in the normal way:

  if _, m := range msgs {
    // Use.
  }

That makes a copy of the struct. This is not supported in general for the opaque API, even though it appears to work for standard use cases. The representation is meant to be opaque.

It's true that this would perform better, and greatly reduce allocations. But:

  - Messages (especially opaque ones) are not supposed to be copied. The
    recommendation is to use `m := &mypb.Message{}`.
  - This would make migrating to use the opaque API more difficult, if the
    getters don't return the same type as the old open API fields, much more
    code needs to be rewritten, or some wrapper that allocates a new slice on
    every get.
  - Users expect that `subm := m.GetSubMessages()[2] ;
    m.SetSubMessages(append(m.GetSubMessages(), anotherm))) ; subm.SetInt(42) ;
    assert(subm.GetInt() == m.GetSubMessages()[2].GetInt())`. This would not be
    the case if the API returned a slice of values.
  - ...

Effectively, a slice of pointers is baked into the API, and the way people use protocol buffers in Go. For these reasons, it's not clear to me this would end up performing better or causing less work.

If we had returned an iterator (new in Go 1.23) instead of an actual slice, then it would've been possible to vary the underlying representation (slice-of-pointers, slice-of-value-chunks, ...). But there are other downsides to that too:

  - Allocations when passing iterators to functions that expect a slice.
  - Extra API surface for modifying the list (append, getn, len, ...).

Not that clear of a win either.

Another thing that could be considered is: when decoding, allocate a slice of values ([]mypb.Message), *and* a slice with pointers (or do it lazily): []*mypb.Message. Then initialize:

  for i := range valuel {
    ptrl[i] = &valuel[i] // TODO: verify that this escape doesn't cause disjoint allocations.
  }

That might be beneficial due to grouping allocations, and the user would be none the wiser.

Can you be specific? I'm curious.

Both, actually. Without accessor methods, laziness couldn't be implemented.

The author disables SMT (hyperthreading) like this:

    disabled_cpus=(1 3 5 7 9 11 13 15)
    for cpu_no in $disabled_cpus ; do
      echo 0 | sudo tee /sys/devices/system/cpu/cpu$cpu_no/online
    done

But there is an easier way on Linux that doesn't require parsing /sys/devices/system/cpu/cpu*/topology/thread_siblings_list:

    sudo tee /sys/devices/system/cpu/smt/control <<< off

You've obviously done a bit of research. Have you ever seen the vitamin B12 link referenced in other studies? Are you taking supplements or did you get it tested?