cosmos0072

Found: it's a sentence from 2020, and PG decided not to appeal (!?)

Full story (in Italian) at https://www.wired.it/internet/web/2020/06/30/progetto-gutenb...

From Italy, https://www.gutenberg.org/ gives a 404 error and https://gutenberg.org/ opens a very official-looking page stating "police notice. This site is under judicial seizure" and references a sentence number: "criminal proceedings 52127/20 R.N.R.I. tribunal of Rome"

Any idea what's happening? I thought PG published public domain books...

> I am not familiar with the Scheme ecosystem but see you are implementing a lot within your shell (http client, sqlite support) what is great.

Yes. Initially, I wanted to integrate http client, sqlite client etc. directly in the main shell executable.

This turned out to be impractical, because it introduces compile-time dependencies on external libraries, and because any bug, vulnerability or crash in the C code would bring down the whole shell.

My current solution is to have separate C programs (currently `http` and `parse_sqlite`) that are not compiled by default: you need to run `make utils` to build them.

> But does that mean I can produce a Schemesh binary that will include Chez Scheme and run my script on any Linux or FreeBSD host?

Yes, exactly. Schemesh is a C program that includes both Chez Scheme REPL and shell features.

Yes, they are similar at first glance. As I wrote:

> If you know nushell, they will feel familiar as they are inspired by it - the implementation is fully independent, though.

Looking deeper, there are two main differences:

- Nushell structured pipelines are an internal construct, and only work with nushell builtins. Schemesh uses actual POSIX pipes, which allows to also insert executables in the pipelines.

- schemesh also allows to insert arbitrary Scheme code in the pipelines. Nushell does too, in a sense: you have to write nushell language though

First stable release appeared on HN one year ago: https://news.ycombinator.com/item?id=43061183 Thanks for all the feedback!

Today, version 1.0.0 adds structured pipelines: a mechanism to exchange (almost) arbitrary objects via POSIX pipes, and transform them via external programs, shell builtins or Scheme code.

Example:

  dir /proc | where name -starts k | sort-by modified

possible output:

  ┌───────────┬────┬───────────────┬────────┐
  │   name    │type│     size      │modified│
  ├───────────┼────┼───────────────┼────────┤
  │kcore      │file│140737471590400│09:32:44│
  │kmsg       │file│              0│09:32:49│
  │kallsyms   │file│              0│09:32:50│
  │kpageflags │file│              0│10:42:53│
  │keys       │file│              0│10:42:53│
  │kpagecount │file│              0│10:42:53│
  │key-users  │file│              0│10:42:53│
  │kpagecgroup│file│              0│10:42:53│
  └───────────┴────┴───────────────┴────────┘

Another example:

  ip -j route | select dst dev prefsrc | to json1

possible output:

  [{"dst":"default","dev":"eth0"},
  {"dst":"192.168.0.0/24","dev":"eth0","prefsrc":"192.168.0.2"}]

Internally, objects are serialized before writing them to a pipe - by default as NDJSON, but it can be set manually - and deserialized when reading them from a pipe.

This allows arbitrary transformations at each pipeline step: filtering, choosing a subset of the fields, sorting with user-specified criteria, etc. And each step can be an executable program, a shell builtin or Scheme code.

If you know nushell, they will feel familiar as they are inspired by it - the implementation is fully independent, though.

Parents already have a lot of control on children' education.

Examples: most children believe in the same religion as their parents, and can visit friends and places only if/when allowed by their parents.

This is simply extending the same level of control to the internet.

Government-mandated restrictions are completely another level.

I am from EU, and contrary to age verification laws in general.

My stance is that if somebody is a minor, his/her/their parents/tutors/legal guardian are responsible for what they can/cannot do online, and that the mechanism to enforce that is parental control on devices.

Having said that, open-source zero-knowledge proofs are infinitely less evil (I refuse to say "better") than commercial cloud-based age monitoring baked into every OS

I need this *so* often that I programmed my shell to execute 'cd ..' every time I press KP/ i.e. '/' on the keypad, without having to hit Return.

Other single-key bindings I use often are:

KP* executes 'ls'

KP- executes 'cd -'

KP+ executes 'make -j `nproc`'

The math looks suspicious to me, or at least how it is presented.

If, as stated, accessing one register requires ~0.3 ns and available registers sum up to ~2560 B, while accessing RAM requires ~80 ns and available RAM is ~32 GiB, then it means that memory access time is O(N^1/3) where N is the memory size.

Thus accessing the whole N bytes of memory of a certain kind (registers, or L1/L2/L3 cache, or RAM) takes N * O(N^1/3) = O(N^4/3).

One could argue that the title "Memory access is O(N^1/3)" refers to memory access time, but that contradicts the very article's body, which explains in detail "in 2x time you can access 8x as much memory" both in text and with a diagram.

Such statement would require that accessing the whole N bytes of memory of a certain kind requires O(N^1/3) time, while the measurements themselves produce a very different estimate: accessing the whole N bytes of memory of a certain kind requires O(N^4/3) time, not O(N^1/3)

For me it's more about "feeling" badly written code, as if it stinks and is unpleasant to look at, and conversely enjoying nicely written one, and seeing its elegance and beauty.

Especially when I have to familiarize with code written by someone else, I usually start by "cleaning" it - small refactorings such as splitting overlong functions and simplifying expressions, that are trivial to prove as correct even without a deep knowledge of the code purpose.

Only when the code looks sufficiently clean and familiar, I start adding the requested new features

> So, people who use native REPLs, what do you do with them?

In my case, I use my interactive shell https://github.com/cosmos72/schemesh every day as login shell.

You can look at it as heavily customized Scheme REPL, where everything not inside parentheses is parsed and executed as shell syntax, and everything inside parentheses is parsed and executed as Scheme syntax.

Having arithmetic and procedure definition within the login shell definitely feels liberating, at least to me

I tried compiling https://github.com/panzi/ansi-img and running it inside the truecolor branch of twin, i.e. https://github.com/cosmos72/twin/tree/truecolor and it correctly displayed an example JPG I tried - no artefacts, and terminal remained responsive.

As I wrote above, I am debugging it and fixing some known issues, thus truecolor support is not yet in the default branch

You can choose the font with `twin --hs=X11,font=X11FONTNAME` or `twin --hw=xft,font=TRUETYPEFONTNAME`

I will have a look too, the terminal should not crash or stop being responsive

If you mean the Unicode glyphs listed at https://en.m.wikipedia.org/wiki/Block_Elements they are supported - you just need a display driver that can render them. For example, `twin --hw=xft` (it's the default) or `twin --hw=X11`, both with a font that contains them

Yep, I am well aware of the `;` vs `:` confusion in both 256 color and 24-bit color control sequences.

Short of hand-coding "which terminal supports which variant" I do not know any standard mechanism to detect that (beyond the well-known $TERM=...-256color and $COLORTERM=truecolor or $COLORTERM=24bit)

I guess I'll have to add command line options to choose among the variants 1...7 you helpfully listed above.

My main use it to render twin directly on X11, which avoids all these issues, and while rendering inside another terminal is important and is not going away, I am OK with a few minor color-related limitations (note: limitations, not bugs) in such setup, especially if the other terminal does not follow the relevant standards

Author here :)

I've been using Twin as my everyday terminal emulator and terminal multiplexer since ~2000, slowly adding features as my free time - and other interests - allowed.

As someone pointed out, the look-and-feel reminds Borland Turbo Vision. The reason is simple: I started writing in in the early '90s on DOS with a Borland C compiler, and I used the Borland Turbo Vision look-and-feel as a visual guideline (never actually looked at the code, though).

The porting to linux happened in 1999 (it was basically dormant before that), and Unicode support was progressively added around 2015-2016 (initially UCS-2 i.e. only the lowest 64k codepoints, then full UTF-32 internally, with terminal emulator accepting UTF-8). There are still some missing features, most notably: no grapheme clusters, no fullwidth (asian etc.) support, no right-to-left support.

Right now I'm adding truecolor support (see https://github.com/cosmos72/twin/tree/truecolor) - it's basically finished, I'm ironing out some remaining bugs, and thinking whether wire compatibility with older versions is worth adding.

And yes, documentation has been stalled for a very long time.

Retrospectively, I should have switched C -> C++ much earlier: lots of ugly preprocessor macros accumulated over time, and while I rewrote the C widget hierarchy as C++ classes, several warts remain.

For optimal speed, you should move as much code as possible outside the closures.

In particular, you should do the `switch op` at https://github.com/skx/simple-vm/blob/b3917aef0bd6c4178eed0c... outside the closure, and create a different, specialised closure for each case. Otherwise the "fast interpreter" may be almost as slow as a vanilla AST walker.

The core idea is simple: do a type analysis on each expression you want to "compile" to a closure, and instantiate the correct closure for each type combination.

Here is a pseudocode example, adapted from gomacro sources:

https://gist.github.com/cosmos72/f971c172e71d08030f92a1fc5fa...

This works best for "compiling" statically typed languages, and while much faster than an AST interpreter, the "tree of closures" above is still ~10 times slower that natively compiled code. And it's usually also slower than JIT-compiled code

Compiling an expression to a tree of closures, and a list of statements to a slice of closures, is exactly how I optimized [gomacro](https://github.com/cosmos72/gomacro) my Go interpreter written in go.

There are more tricks available there, as for example unrolling the loop that calls the list of closures, and having a `nop` closure that is executed when there's nothing to run but execution is not yet at the end of the the unrolled loop.

You mean a command or builtin `(` inside a traditional shell as bash or zsh?

It would be quite limited:

internal status would not persist between invocations,

and it would only be able to exchange unstructured data (a stream of bytes) with the shell

That's true.

In shells, "test" and "[" are often used after "if", as for example

  if test -f "some_file"
    do_something
  fi

  if [ "$FOO" != "" ]
    do_something_else
  fi

Schemesh does not have a shell builtin "if", you switch to Scheme for that:

  (if (file-regular? "some_file')
    (sh-run {do_something}))

Thus the need for "test" and its alias "[" is reduced.

Also, "test" implements a mini-language full of one-letter operators: `-f FILE` `COND1 -a COND2` `COND1 -o COND2` etc.

I really don't miss it, as I find the equivalent in Scheme to be more readable - and of course more general

  (file-regular? "FILE")
  (and COND1 COND2)
  (or COND1 COND2)

etc.

[UPDATE] (shell-expr) is ready and kicking :)

Now you can write

  (import (schemesh all))

  (shell-expr (lisp-expression-writing-to-sh-fd-stdout)) | some-shell-command-and-args | (shell-expr (lisp-expression-reading-from-sh-fd-stdin))

An example of such expressions is: For writing to sh-fd-stdout,

  (fd-write (sh-fd-stdout) (string->utf8b "hello\n"))

For reading from sh-fd-stdin,

  (display (utf8b-bytespan->string (fd-read-until-eof (sh-fd-stdin))))

Yes, that's definitely feasible.

I am currently working on it, the macro will be named (shell-expr) and replace the current experimental (shell-test)

Porting to a different Scheme implementation requires some effort: schemesh needs a good, bidirectional C FFI and an (eval) that allows any Scheme form, including definitions.

For creating a single `schemesh` executable with the usual shell-compatible options and arguments, the Scheme implementation also needs to be linkable as a library from C:

Chez Scheme provides a `kernel.o` or `libkernel.a` library that you can link into C code, then call the C functions Sscheme_init(), Sregister_boot_file() and finally Scall0(some_scheme_repl_procedure) or Sscheme_start()

Nice! then my point 3. above is wrong and should be deleted

Rash and schemesh start from similar ideas: create a shell scriptable in some dialect of Lisp.

Rash has several limitations, sometimes due to design choices, that schemesh solves:

1. no job control

2. multi-line editing is limited

3. from what I understand, shell syntax is available only at REPL top level. Once you switch to Lisp syntax with `(`, you can return to shell syntax only with `)`. Thus means you cannot embed shell syntax inside Lisp syntax, i.e. you cannot do `(define j {find -type f | less})`

4. shell commands are Lisp functions, not Lisp objects. Inspecting and redirecting them after they have been created is difficult

5. Rash is written in Racket, which has larger RAM footprint than schemesh running on vanilla Chez Scheme: at startup, ~160MB vs. ~32MB

6. Racket/Rash support for multi-language at REPL is limited: once you do `#lang racket`, you cannot go back to `#lang rash`

I found another, possibly simpler solution.

The functions (sh-fd-stdin) (sh-fd-stdout) and (sh-fd-stderr) return the integer file descriptors that a schemesh builtin should use to perform I/O.

With them, you can do

  true (lambda () (lisp-expr-writes-to-fd (sh-fd-stdout))) | grep foo | true (lambda () (lisp-expr-reads-from-fd (sh-fd-stdin)))

It should work :)

[REWRITTEN FOR CLARITY]

Yes, although the current syntax is cumbersome - I am thinking how to improve it.

The first part is easy. If you want to run something like

    (lisp-expr1-produces-a-string) | grep foo

the current solution is

    echo (lisp-expr1-produces-a-string) | grep foo

The second part, i.e. feeding a command's output into a Scheme function, is more cumbersome.

If you want to run

   echo (lisp-expr1) | grep foo | (lisp-expr2)

the current solution requires (sh-run/string job), namely:

    (lisp-expr2-accepts-string-arg
      (sh-run/string
        {echo (lisp-expr1) | grep foo}))

If instead you have a (lisp-expr2...) that reads from an integer file descriptor passed as argument - not a Scheme I/O port - you can write

    (lisp-expr2-accepts-integer-file-descriptor
      (sh-start/fd-stdout
        {echo (lisp-expr1) | grep foo}))

[UPDATE] There is also a function (sh-redirect job redirection-args ...) - it can add arbitrary redirections to a job, including pipes, but it's quite low-level and verbose to use

Eshell looks really powerful :)

Does it also have job control, and jobs as first-class objects?

In schemesh, you can do things like

    find / -xdev -type f | ls -l --sort=size
    CTRL+Z
    bg 1

and also

    (define j {git log})
    (display j)
    (display (sh-run/string j))

It does not sacrifice sub-shell syntax: it is fully supported, I just had to rename it from ( ... ) to [ ... ] to avoid conflicts with ( ... ) that switches to lisp syntax

Also, both $(...) and `...` shell interpolation are fully supported.

The only thing I intentionally sacrificed is shell flow control: schemesh shell syntax does not have the builtins 'case' 'for' 'if' 'while' etc.

In practically all examples I tried, escaping to Scheme for loops and conditional code works better: it avoids the usual pitfalls related to shell string splitting, and usually results in more readable code too, at least for a Lisper

Note: there's also some additional parsing logic to distinguish between sub-shell syntax [ ... ] and wildcard patterns that use [ ... ] as well