Org-mode parser in Rust
github.com
github.com
> Why reinventing the wheel when we can just copy it! This project takes the only surefire way to get it right - use the original elisp parser implementation as a blueprint!
Emacs Lisp code is under GPLv3 license, is it OK to rewrite it in another language (using original code as blueprint) and publish it under MIT?
How derivative is defined? Manual compilation is ok, but automatic is not? But what if I beleive that humans are just fancy machines? ;)
> How derivative is defined?
For the US, it's primarily defined in plain English in the USC. But subtle distinctions are made by court rulings/opinions.
You may think that's a strange interpretation, but before GCC many compilers took that exact same stance. You bought the compiler and you did not have a license to distribute the resultant binary. You had to pay an extra fee for the right to distribute your own compiled code. That pretty much stopped when Borland released a compiler with a mostly sane license. You can imagine why, though, that Cygnus made so much money doing contract work for GCC in the embedded space -- every other compiler required royalties for everything you shipped!
rewriting the parser to make a background image for a desktop could be derivative as being "artistic" is not anywhere near the intention of the original creation. So the picture could be MIT and the code the picture shows GPL.
In general how you use something is more relevant than how you produce something. A human translation with heavy restructuring of the code and improvements can be derivative (maybe) putting it inside a pipeline (be it a transpiler or a contractor) wouldn't.
https://www.gnu.org/software/bison/manual/html_node/Conditio...
It is not according to their FAQ.
> If the original program is licensed under certain versions of the GNU GPL, the translated program must be covered by the same versions of the GNU GPL.
As I recall, Stallman used to do something similar early on pre-GNU and stopped when he realized it wasn’t technically necessary, wasn’t even particularly helpful to himself in particular, and might even have legal implications to reimplement software with that method.
Wow. I'm really curious about what legally counts as a translation. It's an interesting mathematical problem.
As I understand it copyright doesn't apply to the algorithm. It does apply to things like code organization, variable naming, and comments.
Seems to be a pretty direct translation (without reading the sources of both).
Now, if they took the original source and used it to implement a grammar then based the rust code off said grammar they might have a case -- which, I believe, is what a "clean room" implementation would do, preferably with two different people doing either task.
IMHO this is a clear copyright violation else you could just "reimplement" your way out of any license.
- Accidentally copying copyrighted material (past what fair use allows)
- Expensive lawsuits where you have to actually argue that "no we didn't copy copyrighted elements"
We're talking about two open source projects here, the latter anyways doesn't really matter.
As someone who GPL-licenses most of his software, this is a good reminder of what it looks like when copyleft licenses work as intended.
No. However if you just take a specification of the code’s expected behavior (in this case a description of the markup language), and do a “clean-room” reimplementation where you never look at the original code, you should be in the clear from what I understand.
But, I think it’s great that more people are looking at org-mode outside emacs. I’d love to see alternative tool chains that push the capabilities and open more people to its magic.
I often see Org and OrgMode conflated, where OrgMode seems to provide much more of an experience with the way Emacs has integrated it into the editor. Could markdown be as integrated and provide similar experience?
Been using it for >2 years now and for sure only using a fraction of its capacity.
The format seems better suited to extension too with ways to have annotations or blocks
# Level 1
## Level 2
### Level 3
#### Level 4
etc.
Level 1
=======
Level 2
-------
One could, in theory, create an extended Markdown flavor that allows similar metadata, but that wouldn't be compatible with any existing tools anyway, so there's not much point.
For example, in Org you can add task management data, scheduling information and arbitrary properties to the outline,
#+COLUMNS: %ITEM %TODO %ASSIGNED_TO %SCHEDULED %PRIORITY
#+PROPERTY: ASSIGNED_TO_ALL juki some other people
* TODO [#A] Some task with priority A
SCHEDULED: <2019-04-20>
:PROPERTIES:
:ASSIGNED_TO: juki
:END:
Some description.
For another example, you can add formulas to a table,
#+CAPTION: A table with formulas.
| foo | bar | Row sum |
|-----+-----+---------|
| 10 | 20 | 30 |
| 30 | 40 | 70 |
|-----+-----+---------|
| 40 | 60 | 100 |
#+TBLFM: $3=vsum($1..$2)::@4=vsum(@I..@II)
It's hard to separate .org the markup language from org-mode the application, because quite a lot of its syntax is driven by application features. TODO markers, time stamps, schedule/deadline markers, property drawers, log books, tags - all of these are part of Org Mode functionality, but have little use if you're just trying to write a document for export.
--
[0] - There's ton of stuff you could optionally add with #+SOMETHING: lines - local configuration of how to work with the file, metadata, exporter settings for any format individually and all of them together, and more. A lot of those can also be specified per-headline.
[1] - https://orgmode.org/manual/Macro-replacement.html; crazy, but when it's needed, it works wonders.
[2] - You can make Markdown-style code spans in Org Mode ~like this~, but there's actually a syntax for proper inline source code blocks that's much less known: src_lang{code}, or src_lang[header-args]{code}. For example, src_lisp{(+ 2 2)} or src_lisp[:results output]{(format t "Foobar")}. The reason for the header-args variant is that Org Mode inline code blocks can be executed interactively or on export, just like regular code blocks. See: https://orgmode.org/manual/Structure-of-code-blocks.html for details on both.
[3] - In CommonMark you could technically abuse HTML for that, but Org Mode provides the most readable format for tables... that's also the most convenient to write if you're using Org Mode, and an absolute PITA if you're not. See: https://orgmode.org/manual/Tables.html.
CommonMark also supports code blocks with language hints (using the ```lang... syntax). What Org-mode has, in addition to code blocks with language hints, is inline code with language hints, which neither CommonMark nor GFM support.
> tables
I'm horrified that, indeed, CommonMark doesn't have tables, though you're right that most people now just use GFM.
- org mode is at least as powerful as excel thanks to gnu cal or straight up lisp
- org mode tables are nuts in the sense that you can refer to cells from your text (paragraphs, titles etc). Its neat because it means that when writting a report, you can refer to results of computations in tables - meaning that you can then change constants in the table and numbers will update in you document body (paragraphs etc)
Ive tried explaining org mode with as little words as possible to people and it goes like this:
- its as simple as markdown (a subset of orgmode is isomorphic to md)
- its as powerful as jupyter notebooks (although you can have a cell in scala refering to the output of a python cell itself taking as inpute the output of a previous bash cell)
- its as powerful as excel and word, but you can refer to table cells into your document (copy paste excel table in word, refer to table cell from word body, change excel and see changes appear in word paragraphs refering to said cells)
- its as poweful as pandoc (my spacemacs exports org files to latex, html, odt, md, etc in 5 keys)
- its a fantastic todo manager (drop todos in documents instead of going to todo app and refer todo to org file) although you git for multi user todos.
- its got macros (so you can make macros that will inject html/tex code at export so you can have arbitrarily complex pdfs (html+princexml+css or just latex). If anyone’s interested, i can write an article about this paricular feature. Ive automated my reports to have double page bg images and other goodies but (macro)-enhancing org)
- its got a UI (aka shortcuts) to easily add tables, rows, lines, realign tables etc. So text tables end up being more productive than word/excel tables (although they’re a pain to git since git will version table “indents” instead of cell content since a line can change but not its semantic content)
- its infinitely hackable (make a gantt program in js and make an org cell that takes some DSL code and outputs gantt. It already exists for uml, flowcharts (in emacs you have artist mode, so text flowcharts are eazzy) etc)
I thinks thats about all it is, but i’ve surely forgot about feature im unaware about because i havent needed them yet.
Org mode surely reminds me of apple UIs which ive always described as “your grandma can use it, yet your avg power user will find his spot”. Org is as easy as Md if that’s what you want, but also as powerful as you need it to be if thats what you want.
Thanks for org mode and emacs, its taught me about the sort of things you can make a computer do. And ive only been an emacsian (?) for 2 years, but as a 27 year old (emacs still conquering youngsters ?), ive finally found what i needed to be productive, and hot damn have i not been these past to years (thanks to org, magit, ace, dired, tramp, flycheck, term, evil (im a vim guy originally), helm, which-key, undo tree, the elisp api, paredit, the many others im forgetting, and of course spacemacs). And originally i switched to emacs to have a better window control, lol. Thanks to vs code (which btw is a fantastic editor) which didnt have what i needed (i could switch from a pane to the tree with a shortcut but i had to click to escape the tree (but with helm no trees needed anymore, fiouf), ive discovered life :)
But yeah, org mode is insanely powerful, give it a try. If i were to think of org mode’s power and flexibility as a text editor, its be emacs ;p (or is it the opposite?)
Cheers,
A GNU fanboy
On surface level, `org` files don't bring carry many significant advantages over similar formats like Markdown; rather, it's Emacs's `org-mode` that brings this text file to life.
You can have a full-fledged calendar and scheduling system just in a few lines of text. Imagine that.
This enables .e.g. transformation of org-mode files with latex templates, to html slides or even into .odt files and customization via filters over pandocs AST. These filters can be not only written in haskell but in pretty much any language.
When I started writing a static site generator powering my current personal site, I wanted to parse .org files in the generator. I very quickly gave up and decided to manually export my .org posts to HTML, and parse that instead (see [0]). I didn't realize back then that Pandoc can handle .org files; I probably would have decided to shell out to it instead. Then again, if there's one thing I learned writing a static site generator (twice), it's that the next time around, I'll pick up one that's already written by someone else :).
--
My main use case is exporting scientific manuscripts from org to semi-workable MS Word docs for collaborators to edit. Not sure how merging changes back in is going to go...
Hence, someone would have to properly clean-room reverse-engineer org-mode in order to achieve 100% compatibility, or make a somewhat-incompatible implementation based on the reference docs instead.
As I understand it, Markdown has significant limitations which org-mode doesn't have, but is under a 3BSD-style license. (It also has issues of x+1 slightly incompatible implementations)
Anyone have a recommendation for an article or video or something on the subject?
Lots of "productivity"-applications are either TODO-lists or meant for note-taking or good for writing down concepts or are knowledge bases.
But you can't really separate these. Often I'll write something down and later realize that I need to do xyx for that. Or that I want to schedule a reminder for this note.
Other times I write something down just to have it written down somewhere and I won't want to see it again until months later. But if I just write it into a random .txt-file, I'll probably never see it again. With it being in the same place as my other notes, todos etc., I will actually find it again.
And other times, you might have this dumb idea for a thing. And then you add some more ideas to this idea. And before you know it, you've written down the entire concept for your next software project in your notes-application. When this happens to me with OrgMode, I really don't mind. Its plaintext-format is just as readable as MarkDown for this use-case. I might very well stick this OrgMode-file into the software repository.
Also, sometimes I will find an article to something in my software project concept and want to note it down there with a reminder to actually look at it. OrgMode allows me to do that.
Eventually this becomes a lot of information in one place, but OrgMode itself has lots of tools to organize that: tags, priorities (from A to whichever letter you like), notebooks (=different files), states (e.g. TODO, DONE, NOTE, and again whatever you like), scheduled times, deadlines, recurring tasks etc.
And then there's obviously also parsers like the one in this post. They can do whatever they want with the plaintext you have there.
Personally, I mostly use the Android-app Orgzly. It has a widget with a simple DONE-button for my TODO-usage. It can throw notifications at me. And it has a really powerful search-feature, allowing to save specific searches and for example select them as the listing to display in the widget. E.g. I only want to see notes with the state "TODO" that are scheduled within the next three days.
I will admit that it's a bit of a rabbit hole, but task/note management in general usually is and this feels better than my previous solutions.
Especially also the fact that it is just plaintext-files that I can sync and backup easily.
Yeah, my current, main ideas txt file is something called "movies to watch.txt" —which does start with a list of movies, but then I added some random extended thoughts there temporarily and it has grown massively since.
So yep, I may be a good candidate for using OrgMode ;)
I tried out a demo of Notion (https://www.notion.so/) not long ago and was pretty impressed by their editor (similar concept to OrgMode, but wysiwyg, which I prefer if it's done well). It's actually really good (annoyingly you have to give an email address to try, and their other info. isn't nearly as informative as just trying it for a minute)—but I can't bring myself to use it because of the platform lock-in aspect. I want the data stored in a local file and that's it.
Edit: looking a little more though (was watching a video of table creation in OrgMode), I think the emphasis on editing plain text is going to be too off-putting for me: I do want a simple local file that multiple programs could work with, but the text editing required for OrgMode seems way too cumbersome unless you're already an emacs devotee—which I'm not (largely because I've had RSI issues, and rumor is emacs isn't the most ergonomic text input style).
If you don't like key-chords then Emacs probably won't work for you, but just in case the video you watched wasn't clear about it, most of the markup is handled pretty much automatically by Emacs. Table creation and editing in particular works pretty much like in a graphical application. For example, if you want to create a table like
| foo | bar | quux | qwerty |
|-----+-----+------+--------|
| 10 | 20 | 30 | 40 |
| 50 | 60 | 70 | 80 |
|foo|bar|quux|qwerty
| foo | bar | quux | qwerty |
|-----+-----+------+--------|
| | | | |
There are typical table editing commands available like moving, deleting and inserting columns or rows with [shift+]alt+arrows.
However, Evil mode now works pretty amazingly right now if you like Vim. I use Emacs with Evil and Vim practically interchangeably. My biggest frustration is a bug (which is probably unfixable) with how undo doesn't always correspond with a vi command. There are some details with how buffers are specified, etc, but you can customise it to get close.
Spacemacs is a pre-setup Vim setup for Emacs using Evil mode which is highly recommended, though I haven't used it (I like minimal setups). I don't know if anyone has ever setup edit-mode key bindings for Org mode, but I just use the Emacs ones (I was originally an Emacs user before switching to Vim... and then back to Emacs :-P )
I've also been thinking for a couple hours now about how much work it'd be for me to write a small app with the minimal feature set that I'd like from org-mode :) This is almost certainly a terrible idea for me to do in the context of the rest of my life right now.
Using Emacs just for org-mode is actually a pretty appealing option to me considering the description you gave. So I may give that a shot. Thanks!
Even as a years-long Vim user it taught me some new tricks.
The awesomeness of Org Mode comes from how all the little pieces come together to support each other, and how the incremental the learning curve is. You can use any subset of features you like, and you don't pay (in terms of noise in your files) for the features you don't need. When you start, it's just a Markdown with superb editor support. Step by step, you may eventually find yourself organizing your life in plain text.
As the documentation says, Org is a toolbox[0]. You use only what you need, but you can do a lot with it, and everything interoperates to create a whole much greater than the sum of parts (which ordinarily would each be a different application). This is a common theme for Emacs itself too, and the reason some people (myself included) increasingly live in it - there are great benefits coming with thorough interoperability, as every small new thing you learn or add improves almost every kind of task you do in Emacs (org mode included).
As for introductions, there are things that'll show up under "introduction to org mode" search query. I guess you could read[1] to get an overview, play around, and then see what interests you. There's plenty of demonstrations and guides for specific use cases of Org Mode on-line[2], and you can always ask Org users to share their workflow (and config files) :).
--
[0] - https://orgmode.org/manual/Summary.html#Summary
[1] - https://orgmode.org/orgguide.pdf
[2] - Some listed here: https://orgmode.org/worg/org-tutorials/.
> Org is probably the best and most complete plain text organizational system known to mankind. It has countless applications like authoring, publishing, task and time tracking, journal, blog, agenda, wiki etc...
Yes, absolutely.
> Unfortunately Org was originally developed for Emacs and therefore available only inside Emacs. It is a huge limiting factor for Org's development and popularization. Because of that it is not as popular outside of Emacs community as it should be.
"Unfortunately"? It is a dynamic tool, I don't understand what it would mean for it available outside of emacs except for if that were also a dynamic tool. Are you going to create a standalone dynamic org mode application? It seems better to make a "stripped down" emacs that only has org mode and literate programming features so as to not have as steep a learning curve as the entirety of emacs all at once. What other reason is there for this work except for avoiding the learning curve of emacs? Emacs has got to be one of the most configurable pieces of software outside of Operating Systems(maybe?) in existence.
Org mode is powerful because it is a plaintext system that gives hooks for emacs abilities like transformation (faces, folding, tagging/searching, agenda features) and execution of code with literate programming, and passing values around between code blocks and so on.
Why would this want to live outside of emacs? I do think the recently (and again two years ago) extremely popular post on org mode's markup language absolutely misses this point[1]. This post is also linked to from the repo.
I'd love to learn about what I'm missing here, and would love to get some answers to my questions or responses to my assumptions! I'm very interested in your roadmap and design decisions, I think your external links (except for that article) are fantastic. I will follow the project for sure.
I just think there is some mental disconnect with how org mode is represented and discussed on hn, and those discussions almost all seem to miss the majority of org mode abilities that absolutely depend on living in the style of environment that emacs provides. From reading the top discussions here, one might think org mode is in competition for the same space as markdown while that is not even remotely true [2].
[1]: https://news.ycombinator.com/item?id=19622019
[2]: https://hn.algolia.com/?query=org%20mode&sort=byPopularity&p...
PS: I am trying to do a more thorough review of the discussion about org mode on hn, and I will be taking actual notes and comment links down to better explain my ideas here rather that just general takeaways. Thanks for bearing with me, or letting me know if you disagree.
Now I'm having experiencing some Baader-Meinhof phenomenon as I've seen two org-mode articles on the front-page here since and never noticed any before.
Anyways, this seems sort of obvious to me as a total org (and emacs) beginner... the sheer flexibility and power from this being built on top of the emacs system is the real killer feature, though lots of other organization apps would do well to copy some the base features. I really don't understand why repeated tasks is so hard for other task management apps to implement well and is yet easily done by whoever put them into Org:
https://orgmode.org/manual/Repeated-tasks.html ...
** TODO Call Father
DEADLINE: <2008-02-10 Sun ++1w>
Marking this DONE shifts the date by at least one week, but also
by as many weeks as it takes to get this date into the future.
However, it stays on a Sunday, even if you called and marked it
done on Saturday.
** TODO Empty kitchen trash
DEADLINE: <2008-02-08 Fri 20:00 ++1d>
Marking this DONE shifts the date by at least one day, and also
by as many days as it takes to get the timestamp into the future.
Since there is a time in the timestamp, the next deadline in the
future will be on today's date if you complete the task before
20:00.
** TODO Check the batteries in the smoke detectors
DEADLINE: <2005-11-01 Tue .+1m>
Marking this DONE will shift the date to one month after today.It allows us to throw different use-case-optimized GUIs at a sync'ed folder of org-mode files. A clear example are the mobile apps, but there are others.
I've been using org-mode for a year, trying to replace a 5 year-old Evernote knowledge base/todo list, which itself inherited from a 12 year-old InfoSelect knowledge base.
For task management, org-mode on spacemacs complemented with beorg on mobile is proving to be the best system I've ever used.
As for the knowledge base features, however, I'm still far from being able to reproduce the speed and precision of recall that I had on InfoSelect, or even Evernote with a solid tag hierarchy. I also use Dynalist to have an outliner on mobile.
On mobile, the KB situations is even worse, since beorg is optimized for the task management use-case and does not provide proper search.
I would also love to be able to browse connections in my knowledge base, TheBrain-style, but that will always be clunky to unusable on a pure-text UI.
It would be awesome if I could get rid of Evernote and Dynalist and just point different apps at the same org file repository.
This is one reason having alternate parsers for the org markup is important.
This is a good point, I agree.
> As for the knowledge base features, however, I'm still far from being able to reproduce the speed and precision of recall that I had on InfoSelect, or even Evernote with a solid tag hierarchy. I also use Dynalist to have an outliner on mobile.
What speed and precision are you missing? It seems that org mode should have the capacity to establish any content connection scheme you could dream up.
> KB situations
What does KB mean?
> I would also love to be able to browse connections in my knowledge base, TheBrain-style, but that will always be clunky to unusable on a pure-text UI.
I don't think this is true. Depending on the kinds of connections you want to browse in your knowledge base there are plenty of ways to transform the ui to show them. That is what I see org mode to be at it's essence, a software tool to allow for transformations of text on screens. I think though, that your point is more about browsing. We should have text-based ui browsers for connections. All a gui like the brain is doing is putting animations and a mouse on top, we can definitely specify things via the keyboard in a number of ways to replace the mouse, and who wants animations to begin with? If the main pull at the end of the day is that GUIs are pretty and text UIs aren't then let me point you here [1].
[1]: https://lepisma.github.io/2017/10/28/ricing-org-mode/index.h...
Knowledge Base
That's what I thought too, but I haven't been able to accomplish it yet.
In InfoSelect I had a tree on the left that I could filter instantly and incrementally while I typed in the full-text search box (imagine org-sparse-tree + helm-org-rifle on steroids). In Evernote I have a double hierarchy of notebooks and tags (vaguely based on The Secret Weapon [1]) that allows me to create a new project tag where I pull together new action notes, meeting notes, and old notes from my 17-yo knowledge base. I'll have anything between 5 and 20 ongoing projects that I can toggle quickly on the left hand navigation bar. The notes are shown in a table where I can click on the header to sort by title, created date, or updated date, so I can find what I need within a few seconds.
I can also do a mixed full text/tag/date search adding filtering criteria as I go. In org, I have to decide beforehand what kind of search I want, and if it's a mixed search the syntax becomes very cumbersome.
With org, I can do none of the above on mobile.
In practice, this means that I have more trouble finding things in my 14 month-old org setup than I do in my 17 year-old knowledge base.
> What does KB mean? "Knowledge Base", sorry for dropping the acronym without definition!
> All a gui like the brain is doing is putting animations and a mouse on top, we can definitely specify things via the keyboard in a number of ways to replace the mouse, and who wants animations to begin with?
I have zero problems with using the keyboard. I've been using vim and editors with vim keybindings since the 90s and my favorite laptop of all time doesn't even have a touchpad. There are cases however where specialized GUI affordances can be helpful. I would say that navigating a huge graph of nodes using mind-map-style partial views is probably one of them. I know there's a brave soul trying to pull this off in Org [2] and I sincerely wish them success.
> If the main pull at the end of the day is that GUIs are pretty and text UIs aren't then let me point you here
Pretty is also important but the default Spacemacs theme is pretty enough for me. Mixing fixed and proportional fonts would be nice, I've had that on my to-do list for a while.
[1] https://thesecretweapon.org/ [2] https://github.com/Kungsgeten/org-brain
Honestly, part of it may be just baiting people :). The plaintext markup parts of Org Mode format are really just a Markdown equivalent; the true value comes from the features provided by Org Mode the application itself, and most of these are what they are precisely because of Emacs environment.
These days, with new Emacs users the story usually is "Come for Org Mode and/or Magit, stay for life"; personally, I love to encourage people to take that first step :).
And since a couple of days I'm getting more posts about org-mode. I think are nice foundations to continue on.
... I think this is the nerdiest title I've ever seen posted to HN :)