fjorde

> To me, I don't find Jq at all intuitive.

Same here. Intuitive is probably the wrong word. I think I'm still looking for the right word to use to describe the idea. I think there was some progress from a fb team that translated english language queries into sql.

I hope there is more progress in the field so that the same people who write the content can search it and use it like a dev would. I think that given a stack of 5K blog posts, any dev can pump out a dozen charts and word graphs or whatever given some time, and it would all look pretty good but probably not have as much value as an editor's summary. But if the editor could also make use of the dev tooling to do more in depth research or produce some new insights, or get to the "important" information more quickly, than that would represent a step in the right direction.

I did probably misstate the overall capabilities of jq and grep compared with groq. I wasn't intending to criticize the product and I'm excited about all the offerings in this space that find a niche or provide wider value. Joins across markdown is a great idea and I'm glad this exists.

I don't know if jq can do joins but getting a list of codeblock languages is a pretty simple task for almost anyone in a text file. For complex queries I agree that you typically need a more expressive query language.

I'm really not judging this syntax based on the 2 examples I've looked at, but I agree that I can mostly read all the words and stuff, but without the comments etc this is very dense and I don't think it's "intuitive". Compared to sql it's interesting, and I hope developers like it. I just am almost automatically revolted by the marketing hype train language of the last 15 years. None of the hype comes close to being accurate and it's just all so misleading.

AWS services are among the most notorious in this regard, having sold an entire class of devs on the notion that aws would "abstract away complexity" only to have devs learning about db indices not at the cost of a slow query execution time for some customers, but $50,000 row-scan bills.

AWS hype beasts have been "abstracting away complexity" for years just to tell the poor sap devs that they need intimate knowledge of query execution plans, for some poorly conceived and managed service aws put together after ripping off some os community.

And the connection to aws is not immaterial. Every query is potentially business critical and also susceptible to tanking the entire business. I don't think we should necessarily try to treat queries as "trivial" or whatever.

grep -rE '```\w+' |uniq seems like the same query across text files to me. it's not "trivial" but at least grep has a man page and people have used it reliably for decades now. Spinning up whatever it takes to get groq going is not the same thing as having widely available tools that just work.

Somehow structuring content in json is a new idea? There are a lot of html to json parsers out there already. jq and grep can do everything his post suggests and require no less or more investment from a dev perspective.

appreciate a lot of the points made in the post but

```

distinct( *["code" in body[]._type] .body[_type == "code"] .language )

```

is "trivial" for no human being

yes, these are all fraught. Any X-to-Y format exchange is a brew/apt miracle in my experience. It's not necessarily important for a LaTeX document to be render-able in all media, but the maths disciplines, for example, should be asking themselves how they want to be able to search for things including formula, algorithms etc, and finding ways to guarantee that a maxima of media support search/find/read operations on their articles. It's not just about how easy is it to input? but it's very similar to "how easy is it to get back out?" it has been a bit of a mismatch in maths re formulae expressions, and how those are represented visually vs. textually.

I agree with both these points. Inter-document references are bad in markdown and wiki style formats. Wikipedia is replete with overlapping and contradictory information that should be pulled from a single source.

Do you have any ideas re link-rot, or underlying assets changing and semantic links becoming invalid? How do you reference a code asset whose name could change, what will not only replace all references in the code but also in the documentation?

First, many thanks to the author for a reasoned and researched post. The editing/mass-adoption sides of the text formatting story vis-a-vis markdown were well told. I think the story of text is important and misunderstood and evaluation brings better understanding. So please take the rest of this as a supplement more than an argument, even if a lot of it is a disagreement about the value and future of markdown and where its utility does and could exist.

I'm not partial to markdown or any flavor of markdown or any other plaintext schema. I appreciate org mode and almost every variety of "readable semantic text". These are all reasonable efforts that provide actual value to practitioners and neophytes alike. There are reasons why a tiny fractional percentage of the world uses org-mode. There are reasons why orders of magnitudes more people have come across markdown, to the point where today the overlap with HTML developers is almost, but obviously not fully, complete. On top of the HTML aware writer population, cms's have propagated a lot of incompatible and often poorly implemented "wysiwyg-md" textareas that have often leave a lot of room for improvement, but still introduce the idea of semantic text, or even store that as html/markdown. In all these ways, markdown has been a bridge between the .txt and the .html for many people. It was always a simplification and an added complexity in that regard, and its sole points of reference were txt and HTML, so digital/web-oriented.

That said, HTML semantics differ from .doc(x) semantics primarily in philosophy and ergonomics. In HTML there is <h1>, which has rules and specifications about it. In a .docx document there is no such "understood hierarchy" philosophically speaking. This represents a major structural advantage from a user and machine standpoint that I believe is the single most important advancement represented by markdown. The ability to render an automatic table of contents based on hash-tags or gather/conform citations and references programatically, has _never_ been a selling point of word or any other editor until recently when markdown starting making this type of feature obvious. People will point out word-based variations on "auto-table-of-contents" but please find me one single school in America or anywhere that teaches students to write the "html/md" way by default, in terms of organization or structure. I believe the primary reason they don't is because MSword is a free-form typesetting machine, and not an editor's tool and not a research tool. I was in academic research and talked to dozens of professors publishing in every domain. None of them had a one-click answer to "this journal rejected me; I need to reformat for another journal". That was always done by hand, by every single academic I've ever met, including in maths and sciences and cs disciplines. At best they're using a latex document, but those are way too heavy for non-maths disciplines, and often people aren't or can't make full use of the features it has to offer and end up reformatting latex for various publications.

> If you think about it, do you own your content less if it’s hosted in a database? ... And is it fair to say that proprietary database technology impinges on the portability of your content? ... > But anyone who has tried to move out of a mature WordPress install knows how little this helps if you’re trying to get away from WordPress.

The author addresses a critical aspect of the markdown document but avoids discussing the implications. Text in MSword is hard to get back out.[^1] Data recovery and searchability is critical. But even "pure html" is locked in a lot of un-indexable un-searchable code. markdown, on the other hand, makes grep, or any indexer-search tool trivial to implement and use.

Imagine any company on earth having five-thousand blog posts worth of content. At the end of the day what is the only format that can be guaranteed "readable" in 5-10-20 years time, when some internal analyst wants to find out what people have been posting? Is it easier for the analyst to spin up a wordpress install from 20?? or read through plaintext files that can be searched any number of ways instantly from a kindle reader?

Content lock-in is only one side of the equation. What are you doing with all that content? What is its value? Are you deriving real value from all the content you are creating? Is that value-creation easier or harder because its in txt or sql? These answers depend on who you hire and what you're trying to do. Mostly txt people have questions and sql people help them find answers, but sql people don't always know what questions to ask. So anyways, txt people need text to formulate questions for sql people to run. But neither of them can do anything with docx.

Which is why markdown is loose lingua-franca for the developer world, and it always had to start this way and be this way and pretending like markdown could ever exist for a txt market without a million developer tools and programs is a joke, but nvm. So anyways, markdown required fundamentally a huge buy-in from the dev community and it was well positioned to do so given its html inheritance, regardless of what latecomers to this domain say in public.

Now that there are tools that do a reasonable job translating markdown to pdf, docx, etc, the reality is that for most academics, markdown should represent a huge shift in writing, from a mostly formatting based experience, to a type-and-print model. The same exact .md file should be able to be used to generate the properly formatted and annotated text for any journal publication. This is still a dream mostly, but it could save researchers thousands of hours over the course of their careers. When considered together with enhanced searchability and citability, something like markdown or a "correctly" structured txt document would provide a lot of cognitive "unload".

Anyways, the market is ripe for replacing word, there are hundreds of great text editing products in all the markets and people are piecing together writing systems that make sense for them. It never made sense for academics and students and business people to be typesetters, because that's a professional's job and requires mostly a designer's eye. People want to put important information in a retrievable format and get it back when they need it. pretty-printing for the teacher was always a dumb exercise in scholastic obeisance, but markdown makes a readable document almost by default.

[^1] A <title> tag is easy to use for a title of a document when searching, indexing, scanning, aggregating, etc. But Word defaults you to the filename.docx, which usually ends in DRAFT-FIANL(2001-232-23-).docx What is the title of word document? HTML has <title> and <h1> but each has a specification, and websites are free to conform to those.e

zinekeller's point notwithstanding, i agree that "installer scripts" is really the last necessary domain of the bash script. If you need a (semi)portable script to get `python` to run your python script, you can't use python, obviously.

Things like docker images and nix builds are sold as huge improvements on this chicken-egg problem, at least from a "keep it all in one language" perspective, but only if you consider a docker script (mostly bash) or a nix config truly not the same thing as a shell script that more or less initiates the same environment.

docker/nix abstraction layers have advantages but aren't always as portable as a shell script. If you need a docker engine, I think you'll still need a bash script to install that...

Nix is possibly even more "single command", often just a curl if i'm not mistaken, but still requires config and shipping a build somewhere, and you still need to run the `curl` command in a shell somehow. i don't see how "just use python" will ever fix the init chicken-egg problem.

Perhaps there is a language or nuance barrier. I "thanked" the parent commenter for sharing their view, did an "internet search" for their publicly listed company to see what kind of jobs they might have available, and then was warned about certs visiting their site... Now I'm having a terrible conversation with you for some reason, but I hope this helps dispel your confusion. Please feel free not to reply at all with anything.

take your own advice. i noticed a cert was invalid or broken and mentioned it to a person who might care or know how to deal with it appropriately. In the meantime the site is less secure than it could presumably be. i'm not a paid consultant for the site so i didn't bother digging in and finding out which policy violation triggered a browser warning, but since you did, you can share it here or you can continue to add zero-value comments.

git? it's a bit heavy handed to ask your spouse and kids to clone into the family recipes, but it's basically what you're looking for.

there are git-based markdown cms's like forestry or pug that provide a framework to do this.

I appreciate the sentiments.

i think the cert for https://cieloconnects.com/ is expired.

im not sure how many threads can be pinned, but there isn't an obvious way to highlight these info dense, semi-coherent threads that can just about be read like a long form piece on the subject.

in some ways the idea is almost like zettelkasten for a twitter user. hashtags were derived from zettelkasten note systems, but twitter never encouraged metadata usage to categorize and sustain narratives, or innovative ways to search intelligently across these ideas. the hashtag's functionality was reduced to a viral agent/barometer and soulless corporate actors believed removing spaces from a slogan was the same thing as meaning something.

im pleased with the large proliferation of note-taking and org cms systems today. i think none of them is the answer but the answer wouldn't come about without them or without the experimentation taking place in the space. Being able to easily and accurately collect information across media and various websites and ship/store/recall it all is going to be an incredible shift, but requires replacing almost all the tools we know or only using them as peripherals.

Journalists who deal in sensitive material should not trust any of that to any third parties to the greatest extent possible. If I were Chinese intel or the FBI, otter would be one of the richest targets imaginable for some of the most prized information on earth, i.e. high-value intel targets spilling secrets in "full-confidence" and divulging information they might never reveal even in court. otter also transcribes medical/psych convos and legal discussions as well, bringing the sum total of what they could "know" about any human, willing or not, to scary levels.

Journalists who are casual or reckless with this kind of data shouldn't be in the business, and companies that don't make these kinds of risks apparent to their user base, or go to lengths to disguise a clear conflict of interest as bad support should not be in that business either.

Besides the content of the thread, which is interesting, you'll notice if you scroll through that the thread is added to periodically, whenever the author finds a piece of information that adds to the theme of the thread. This is an underused but valuable way to use twitter. @grayconolloy maintains several of these threads and builds up arguments that can contain years worth of contributing information.

While any thread could be curated to make it look more prescient or more predictive or more correct generally, these threads do demonstrate the overall effect of being able to compile a realtime and longterm "story".

Twitter's most innovative or emergent uses never managed to be useful for most users, and the canned attempts to deliver what people invent have been unsuccessful for the most part. The main reason why to my mind is that it takes literally maintaining a list somewhere of "long-term threads", and remembering which posts to post in which thread. No one does this, but for many in the news-media this would be a valuable way of piling up information into a coherent narrative that people could always "unroll" or whatever.

Please comment with the obvious twitter features that I'm not mentioning. I really don't use it very much anymore and nothing recently has jumped out at me either as a feature or as a feature that people use and promote that replicates this functionality.