Unix settled for string as a data structure and it's little too low level. All Lisp programmers know that s-expressions would have been the right choice. JSON works also.
Even smaller companies I’ve seen end up with an “interfaces” or “types” repo that contains definitions of every external object.
Narrow Waists Can Be Interior or Exterior: PyObject vs. Unix Files - https://www.oilshell.org/blog/2023/06/narrow-waist.html
The appendix links to: Why is Python the language of machine learning?
https://news.ycombinator.com/item?id=29171519
In particular, I highly recommend this bit of history from NumPy creator Travis Oliphant:
https://www.oilshell.org/blog/2021/07/blog-backlog-1.html#co...
Files had structure on pre-Unix OSes, but they don't on Unix, because it doesn't compose.
The "Uniform Interface Constraint" of REST is the same thing, and resembles a file system -- everything is GET / POST on URLs.
---
So the reason that shell/Unix and HTTP are so common is a MATHEMATICAL property of software growth.
How do you make a Zig program talk to a Mojo program? Probably with a byte stream.
What about a Clojure program and a Common Lisp program? Probably a byte stream. (Ironically, S-expressions have no commonly used "exterior" interchange format)
Every time a new language is introduced, I think "well there's another reason you're going to need a shell script".
---
The larger the system, the more heterogeneous it is. And software is larger now, which is why MORE GLUE is needed.
This is why shell was the #6 fastest growing language on Github in 2022: https://octoverse.github.com/2022/top-programming-languages
And the #1 fastest growing language is HCL, which is a very closely related form of glue. It's basically "shell for distributed systems"
---
https://www.oilshell.org/ has JSON as of a few months ago, and is now pure native code (no more Python)
All the normal shell stuff works:
osh$ ls */*.py | wc -l; whoami
327
andy
osh$ var d = {name: 42}
osh$ json write (d)
{
"name": 42
}
shopt --set ysh:upgrade
I think of the data languages as layered -- i.e. an entire json message literally is a string, and it can also contain strings:
https://www.oilshell.org/blog/2023/06/ysh-design.html#exteri...
I mean, I love the look of it, but how can it help me from day dot?
1. you already have a bash script (e.g. in your CI)
2. you want to use an array or dict
You can upgrade to YSH, and start using our Python/JS like arrays and dicts instead. And then you can write them to JSON and upload them somewhere.
---
Another situation is if you use ShellCheck, but you want even more strictness at RUNTIME, then you can do
shopt --set strict:all
Probably because the details of S-expressions differ underneath the hood - a simple (foo ()) can mean a different thing on CL (where the second element is the symbol NIL) than on Scheme (where it is an empty list and not a symbol); also because Clojure introduces additional bracketing patterns, such as [] for arrays and {} for maps, which will confuse non-Clojures. And let's not talk about sexp variants of Lisp-Flavored Erlang.
Also, if you squint hard enough, https://en.wikipedia.org/wiki/Canonical_S-expressions works.
Also I have no idea if anyone ever cared enough to implement cross-language software transactional memory that would work across a process boundary, to close the gap between a Clojure process and a Lisp process. It could work in theory, but someone would need to do it - either get paid to do it, or have enough mojo of their own to write and bulletprove such a thing.
And, since everyone serializes to JSON anyway, serializing to JSON is a decent enough choice for communicating across Lisps as well.
Narrow Waists Can Be Interior or Exterior: PyObject vs. Unix Files
https://www.oilshell.org/blog/2023/06/narrow-waist.html
However the "exterior" extension seems obvious to me -- why hasn't anyone produced a distributed Lisp?
Well I guess Clojure/EDN is that, but nobody has produced a POLYGLOT system involving distributed Lisp. Common Lisp users probably don't use EDN very much, I gather.
---
I found this POSE exterior s-expression format, but it seems to have little interest from the Lisp community:
https://lobste.rs/s/lwf4jv/pose_portable_s_expressions_pose_...
It seems like JSON will have an "impedance mismatch" with Lisp data structures, but yeah that's the cost of interoperability
How about Lisp Flavored Erlang?
I've noticed Erlang/Elixir also favor interior composition! Parsing is not idiomatic there -- they're actually BAD languages for parsing text. (Though they do have good affordances for parsing bytes)
They prefer to pass around Erlang terms, which are copied inside the VM.
I get that -- it's more convenient. But it's also a reason why "the rest of the world" has more code reuse -- e.g. Go talks to JavaScript talks to Python.
Most projects have multiple languages. Monoglot projects become polyglot projects when they grow bigger.
---
So I say
- Lisp prefers its own (interior) narrow waist
- Erlang prefers its own (interior) narrow waist -- even the Lisp one is second class!
- The rest of the world uses the (exterior) byte stream narrow waist, but they complain about it a lot :)
Common Lisp can read and write s-expressions from files. It can also compile source files and load files (both source and compiled).
The syntax for Common Lisp is standardized. Thus a lot of data may be shipped either in text or compiled files. Several implementations can also create snapshots of the runtime memory. A few domains use CL s-expressions (or a subset of that) as language syntax or as a data syntax.
Distributed Lisp applications exist, but may not use text to exchange data. For example there is a defined CORBA mapping for Common Lisp, which enabled Common Lisp software to take part in distributed software using a standard, which allows mixed language software.