Most of the commands are pretty old and do not change anymore. Many parsers are not even commands but standard filetypes (YAML, CSV, XML, INI, X509 certs, JWT, etc.) and string types (IP addresses, URLs, email addresses, datetimes, etc.) which don't change or use standard libraries to parse.
Additionally, I get a lot of support from the community. Many new parsers are written and maintained by others, which spreads the load and accelerates development.
If you read further down in the documentation, you can just prefix your command with `jc` (e.g. `jc ls`). The `--cmd` param is actually a good idea, since it allows you to mangle the data before converting it (e.g. you want to grep a list before converting it).
Regarding maintenance, most of the basic unix commands' output shouldn't change too much (they'd be breaking not only this tool but a lot of scripts). I wouldn't expect it to break as often as you imagine, at least not because of other binaries being updated.
In 20 years, assuming some semblance of moore’s law still holds for storage/RAM/gpu, I’m right there with you.
Is there a good fine-tuning workflow with ollama?
Solution: throw a high end GPU with 24GB RAM and a million dollar of training at it.
Yeah, great solution.
Also, an LLM trained to be good at this task has many more applications than just turning command output into structured data. It's actually one of the most compelling business use cases for LLMs
You have a small C program that processes this data in memory, and dumps it to stdout in tabular text format.
Rather than simplify by stripping out the problematic bit (the text output), you suggest adding a large, cutting-edge, hard to inspect and verify piece of technology that transforms that text through uncountable floating point operations back into differently-formatted UTF8.
It might even work consistently (without you ever having 100% confidence it won't hallucinate at precisely the wrong moment).
You can certainly see it being justified for one-off tasks that aren't worth automating.
But to shove such byzantine inefficiency and complexity into an engineered system (rather than just modify the original program to give the format you want) offends my engineering sensibilities.
Maybe I'm just getting old!
I’d challenge that. Try working with your upstream. It’s easier than ever nowadays to submit issues and PRs on GitHub.
Building layers upon layers, just work around minor issues in a tool is not wise.
Anyway, I do try, but in my experience, if it happens, it's not over night and your going to have to maintain a work around for an amount of time
Maybe the best middle ground is to have an LLM write the parser. Lowers the development cost and runtime performance, in theory
This is not a serious suggestion
Developers make mistakes too, so there are no guarantees either way. Each of your questions can be asked of handwritten code too
It's not a question of "is the output always correct". Nothing is so binary in the real world. A well hand-maintained solution will trend further towards correctness as bugs are caught, reported, fixed, regression tested, etc.
Conversely, you could parse an IP address by rolling 4d256 and praying. It, too, will sometimes be correct and sometimes be incorrect. Does that make it an equally valid solution?
there are plenty of programmers who do not know how to write lexers, parsers, and grammars
As I said before, I maintain a project like this. I also happen to work for a company that specialises in the use of generative AI. So I’m well aware of the power of LLMs as well as the problems of this very specific domain. The ideas you’ve expressed here are, at best, optimistic.
by the time you’ve solved all the little quirks of ML you’ll have likely invested far more time on your LLM then you would have if you’d just written a simple parser and, ironically, needed someone far more specialised to write the LLM than your average developer.
This simply isn’t a problem that needs a LLM chucked at it.
You don’t even need to write lexers and grammars to parse 99% of application output. Again, I know this because I’ve written such software.
> Be kind. Don't be snarky. Converse curiously; don't cross-examine. Edit out swipes.
> Comments should get more thoughtful and substantive, not less, as a topic gets more divisive.
> Eschew flamebait. Avoid generic tangents. Omit internet tropes.
The latest "hammer" is AI.
Lots of commenters here are suggesting to use a complex AI to solve simple text parsing. Maybe you can't see the problem with that, but it's like using 1000 Watts of power to solve something that should take 1 microwatt, just because "new, shiny" AI is here to save us all from having to parse some text.
I'm not making assumptions about what people are commenting about in this thread. Your comment comes off like a subtle troll.
What rule applies when the initial comment is not thoughtful and substantive?
¯\_(ツ)_/¯
<rant>
Since I started programming in the 80s, I've noticed a trend where most software has adopted the Unix philosophy of "write programs that do one thing and do it well". Which is cool and everything, but has created an open source ecosystem of rugged individualism where the proceeds to the winners so vastly exceeds the crumbs left over for workers that there is no ecosystem to speak of, just exploitation. Reflected now in the wider economy.
But curated open source solutions like jc approach problems at the systems level so that the contributions of an individual become available to society in a way that might be called "getting real work done". Because they prevent that unnecessary effort being repeated 1000 or 1 million times by others. Which feels alien in our current task-focussed reality where most developers never really escape maintenance minutia.
So I'm all in favor of this inversion from "me" to "we". I also feel that open source is the tech analog of socialism. We just have it exactly backwards right now, that everyone has the freedom to contribute, but only a select few reap the rewards of those contributions.
We can imagine what a better system might look like, as it would start with UBI. And we can start to think about delivering software resources by rail instead of the labor of countless individual developer "truck drivers". Some low-hanging fruit might be: maybe we need distributions that provide everything and the kitchen sink, then we run our software and a compiler strips out the unused code, rather than relying on luck to decide what we need before we've started like with Arch or Nix. We could explore demand-side economics, where humans no longer install software, but dependencies are met internally on the fly, so no more include paths or headers to babysit (how early programming languages worked before C++ imposed headers onto us). We could use declarative programming more instead of brittle imperative (hard coded) techniques. We could filter data through stateless self-contained code modules communicating via FIFO streams like Unix executables. We could use more #nocode approaches borrowed from FileMaker, MS Access and Airtable (or something like it). We could write software from least-privileges, asking the user for permission to access files or the networks outside the module's memory space, and then curate known-good permissions policies instead of reinventing the wheel for every single program. We could (will) write test-driven software where we design the spec as a series of tests and then AI writes the business logic until all of the tests pass.
There's a lot here to unpack here and a wealth of experience available from older developers. But I sympathize with the cognitive dissonance, as that's how I feel every single day witnessing the frantic yak shaving of "modern" programming while having to suppress my desire to use these other proven techniques. Because there's simply no time to do so under the current status quo where FAANG has quite literally all of the trillions of dollars as the winners, so decides best practices while the open source community subsists on scraps in their parent's basement hoping to make rent someday.