h4l

That model still works for streaming. You have a central source stream only to the distributed edge locations, then have clients only stream from their local edge location. Even if one region is overwhelmed, the rest can still work. Load on the central source is bounded.

Humans have a massive capacity to vary energy use. Highly trained endurance athletes like professional road cyclists and triathletes can average 3x or more the typical daily energy expenditure of a non-athlete on a long term basis. The idea that psychological stress can overwhelm the body's ability to produce energy does not seem credible to me.

I'm really happy to see this, I love using the ipython console and notebooks, and while the deno repl was ok, it's not as good.

I wonder if Deno can be used with https://quarto.org/ now?

I'd be interested in knowing more about your API use. Are you using the official 3rd party API or one of their internal APIs used by their own web front-end?

It's a good idea for a project like this. One bit of feedback is that it would be helpful to have a bit more context for the images - the titles get elipsised on mobile and when viewing a full image you can't see the title.

Thanks! That's right, it's pure Python. It's certainly a lot slower than C or Rust, but I've not done any benchmarking to get a good sense. I believe Deno have a Rust implementation which they use, so in principle a Python binding could make use of that. I think V8's own C++ implementation is too integrated with V8 itself to be re-usable, at least without refactoring.

My immediate use case is very IO-bound, and won't use huge message sizes, so decoding/encoding performance probably isn't a huge problem. My hunch is It should be fast enough for event handling small messages, or it'd also be fine for passing binary buffers between Python and JS. (E.g using an array library like numpy and shipping an array over as a buffer, with some other JS objects for extra metadata.) (More so if I implemented reading/writing an mmap.)

MappingProxyType is another handy one. It wraps a regular dict/Mapping to create a read-only live view of the underlying dict. You can use it to expose a dict that can't be modified, but doesn't need copying.

https://docs.python.org/3/library/types.html#types.MappingPr...

Awesome, thanks for packaging it!

> Is this tool not an additional dependency?

It is, but if you already have bash, adding another shell script isn't much of a jump. e.g. I'd feel OK about committing jb to another repo for use from a .envrc file to set up an environment, whereas committing a binary would not feel good.

> Biggest crime of the Unix world probably.

Sorry if I'm perpetuating this! :) My take is that problem is not with bash, the problem is that it's hard for more advanced tools to replace it.

jc has many parsers for the specific output format of various programs, it can automatically create a JSON object structure using its knowledge of each format.

jb doesn't have high-level knowledge of other formats, it can read from common shell data sources, like command-line arguments, environment variables and files. It gives you ways to pull several of these sources into a single JSON object.

jb understands some simple/general formats commonly used in shell environments:

- key=value pairs (e.g. environment variable declarations, like the `env` program prints

- delimited lists, like a,b,c,d; (but any character can be the delimiter) including null-delimited (commonly used to separate lists of file paths)

- JSON itself — jb can validate and merge together arrays and objects

You can use these simple sources to build up a more complex structure, e.g. using a pipeline of other command line tools to generate null-delimited data, or envar declarations, then consuming the program's output via process substitution <(...). (See the section in the README that explains process substitution if you're not familiar, it's really powerful.)

So jb is more suited to creating ad-hoc JSON for specific tasks. If you had both jc and jb available and jc could read the source you need, you'd prefer jc.

That is pretty cool, and I wish such features were common in regular UNIX shells.

For good measure, this is how you might do the same with jb:

    $ jb Hello=world array:number[]@<(seq 10) object:json@<(date=$(date -Iseconds) jb @date)
    {"Hello":"world","array":[1,2,3,4,5,6,7,8,9,10],"object":{"date":"2024-07-03T19:26:36+00:00"}}

Alternatively, using the :{} object entry syntax:

    jb Hello=world array:number[]@<(seq 10) object:{}=date=$(date -Iseconds)
    {"Hello":"world","array":[1,2,3,4,5,6,7,8,9,10],"object":{"date":"2024-07-03T19:30:26+00:00"}}

Two main situations I think. The first is just interactive use in any shell to encode ad-hoc JSON. If you have a next-gen shell which can handle structured data directly, then you probably don't need it.

Second is situations where you'd rather not add an additional dependency, but bash is pretty much a given. For example, CI environments, scripts in dev environments, container entrypoints. Or things that area already written in bash.

I don't advocate writing massive programs in bash, for sure it's better to turn to a proper language before things get hairy. But bash is just really ubiquitous, and most people who do any UNIX work will be able to deal with a bit of shell script.

Good question! Yep, jb exits with non-zero:

    $ jb size:number=oops; echo $?
    json.encode_number(): not all inputs are numbers: 'oops'
    json(): Could not encode the value of argument 'size:number=oops' as a 'number' value. Read from inline value.
    ␘
    1

If you pipe the jb error into jq, jq fails to parse the JSON (because of the Cancel ctrl char) and also errors:

    $ jb size:number=oops | jq
    json.encode_number(): not all inputs are numbers: 'oops'
    json(): Could not encode the value of argument 'size:number=oops' as a 'number' value. Read from inline value.
    parse error: Invalid numeric literal at line 2, column 0

    $ declare -p PIPESTATUS
    declare -a PIPESTATUS=([0]="1" [1]="4")

So jq exits with status 4 here.

How did you guess my password?!?!

This is just the kind of use case I had in mind. Something I've considered is publishing a mini version with only the json.encode_string function, as that's enough to create an array of JSON-encoded strings and use a hard-coded template with printf to insert the JSON string values.

That would be a fraction of the overall json.bash file size.

There is, the earliest version I've tested with is 4.4.19, but ideally a 5.x version. 3 certainly won't work I'm afraid. If you use homebrew on Mac it's a good way to get the latest bash.

Thank you, that's high praise! I learnt a lot about bash writing this, but I've also not looked at the code in a few months, and it's already starting to look quite intimidating!

I definitely like the idea of a goland/rust implementation, there are certainly things I could improve.

So the argument syntax escapes by repeating a character rather than backslash. I chose this because with backslashes escapes it would be unclear whether a backslash was in the shell syntax or the jb syntax, and users may end up needing to double escape backslashes, which is no fun! Whereas a shell will always ignore two copies of a character like =:@.

The downside of double-escaping is that the syntax can be ambiguous, so sometimes you need to include the middle type marker to disambiguate the key from the value. But the type can be empty, so just : works:

    $ jb ===msg==:==hi=
    {"=msg=":"=hi="}

In the key part, the first = begins the key, the == following are an escaped =. The first = following the : marks the value, and everything after is not parsed, so =hi= is literal.

When you have reserved characters in keys/values (especially if they're dynamic), it's easiest to store the values in variables and reference them with @var syntax:

    $ k='=msg=' v='=hi=' jb @k@v
    {"=msg=":"=hi="}

Powershell has the upper hand here!

Still, bash can try to keep up using json.bash. :)

    $ source json.bash
    $ declare -A greeting=([Hello]=World)
    $ json ...@greeting:{}
    {"Hello":"World"}

... is splatting the greeting associative array entries into the object created by the json call.

Without the ... the greeting would be a nested object. Probably more clear with multiple entries:

    $ declare -A greeting=([Hello]=World [How]="are you?")
    $ json @greeting:{}   
    {"greeting":{"Hello":"World","How":"are you?"}}

Vs:

    $ json ...@greeting:{}                                
    {"Hello":"World","How":"are you?"}

If it helps, there's a little example of using the bash API with bash variables/arrays, should give you an idea of how it could be to use: https://github.com/h4l/json.bash/blob/main/examples/notify.s...

This example uses the pattern of setting an out=varname when calling a json function, the encoded JSON goes into $varname variable. This pattern avoids the overhead of forking processes (e.g. subshells) when generating JSON.

Otherwise you can use the more normal approach of jb writing to stdout, and capturing the output stream.

Thanks for giving it a try and your feedback. I agree, the array splitting is a bit fiddly. It is actually possible to pass JSON directly, you use the :json type on the argument:

    $ jb dependencies:json='["Bash","Grep"]'
    {"dependencies":["Bash","Grep"]}

    $ jb foo=bar bar:json='"this is a well formed string"'
    {"foo":"bar","bar":"this is a well formed string"}

And then you can indeed use command substitution to nest calls:

    $ jb foo:json=$(jb bar=baz)
    {"foo":{"bar":"baz"}}

It works even better to use process substitution, this way the shell gives jb a file path to a file to read, and so you don't need to quote the $() to avoid whitespace breaking things:

    $ jb foo:json@<(jb msg=$'no need\nto quote this!')        
    {"foo":{"msg":"no need\nto quote this!"}}

Another option is to use jb-array to generate arrays. (jb-array is best for tuple-like arrays with varying types):

    $ jb dependencies:json@<(jb-array Bash Grep)
    {"dependencies":["Bash","Grep"]}

And if you use it from bash as a function, you can put values into a bash array and reference it:

    $ source json.bash
    $ dependencies=(Bash Grep)
    $ json @dependencies:[]   
    {"dependencies":["Bash","Grep"]}

I did find it quite satisfying to coerce bash into doing this while maintaining decent performance. I definitely came to appreciate some aspects of bash more from this, but it's so easy to shoot yourself in the foot!

If I started from scratch now I'd use a compiled language that could produce a single static binary and start with really low latency. I'm pretty sure jo must not be tuned for startup time, if they optimised that they must be able get it way faster than bash can start and parse json.bash. I was pretty surprised that bash can startup faster!

The codebase is basically at the limit of what I'd want to do with bash, but there are features I could add if it was in a proper programming language. e.g. validating :int number types, pretty-printing output, not needing the :raw type to stream JSON input.

Thanks for the heads up on Shellcheck, I'd be happy to take a PR if you'd like to.

Glad to hear, this was something I wanted to make reliable, ergonomic and intuitive. I figured a lot of languages use `: type` to declare types.

The same using jo would be like this, which I find harder to type and remember:

  jo -- -s id=42 -n size=42 -s surname=null data=null
  {"id":"42","size":42,"surname":"","data":null}

Notice that surname comes out as the empty string though, I think this must be a bug in jo!

It's nice to have compact single-file dependencies like this! I like the look of your assertions (checking out, err & status). I definitely found myself writing my own assertions to get understandable errors.

Thanks!

As well as anyone's general thoughts/experiences, I'd appreciate opinions on the error handling mechanism jb uses to detect errors in upstream jb processes that jb is reading from.

Normally, detecting errors on the other end of a pipe requires care in a shell environment (e.g. retrospectively checking PIPESTATUS). I used an approach I've called Stream Poisoning. It takes advantage of the fact that control characters are never present in valid JSON. When jb fails to encode JSON, it emits a Cancel control character[1] on stdout. When jb encounters such a character in an input, it can tell the input it's reading from is truncated/erroneous. This avoids the typical problem of a pipe silently being read as an empty file.

I've got a page explaining this with some examples here: https://github.com/h4l/json.bash/blob/main/docs/stream-poiso... I can imagine using control characters in a text stream being rather controversial, but I feel it works quite well in practice.

[1]: https://en.wikipedia.org/wiki/Cancel_character

In the UK our native squirrels are red, but they're almost extinct because North American grey squirrels out-compete them.

You've probably accumulated a lot of relevant html knowledge which will help you.

I presume you have some css-related projects/tasks in mind that you'd like to be able to tackle yourself. I would pick one of those and jump in at the deep end. Although a small goal like restyling something that already works is probably better than starting from scratch.

For a completely abstract task, you could try developing a stylesheet for markdown html output.

Vanilla CSS is good enough (and browser support consistent enough) now that you don't need to use any compilers/frameworks to get started.

If you find yourself struggling for design inspiration, try to re-produce an existing design without seeing how they did it.

I would probably start by opening the browser dev tools on web pages and use inspect element to see how an element in the page has been made to look the way it is. You can edit properties to see what effect they have when changed.

The domain of styling html is so large that there's no chance of learning everything in theory then applying it, I think it's best to learn by doing.

I can imagine it's very hard if you don't know how to use browser dev tools to see the style properties being applied to elements.

Looks like a handy resource you're building here, thanks for this.

FYI the category link right under the title links to a different category to the link below, and the first link only contains one post.

As modern xquery and xslt support JSON, maybe there's scope for an x* tool like jq that makes working with JSON pleasant?

I've not used xquery enough to know if it can be succinct enough to be used as jq's language can.

For sure the Saxon cli could be made a lot more user friendly if it followed normal conventions.

I'll add to what the others have said with an example. The scenario at the start of the article is an example of passive aggression, but it's quite subtle and not explained until later on. Jim has made the effort of cooking pasta for him and Aaron. Carbonara in particular deteriorates quite quickly, so after 10 minutes it won't be very good. We see from the story that Aaron is stuck on the phone with work, and once he finally breaks free, the pasta is spoiled and Jim rather annoyed.

Where's the passive aggression? Later on, the article hints that before this played out, Aaron was annoyed at Jim. He intentionally spoiled the dinner (e.g disrespecting the effort Jim invested) by staying on the phone, but because he pretended to be unable to avoid the call it's passive — he can claim not to have spoiled it intentionally.

In contrast, it would be regular aggression if he'd thrown the pasta into the sink.