A Scholarly Markdown | Hacker News Reader

A Scholarly Markdown

scholarlymarkdown.com

177 pointsdroque96 comments

And for everything else, there's AsciiDoc.[0]

For an extra five minutes learning you get a boatload (think container ship) more features[1] - it compiles to DocBook: a mature, actually standardised, highly structured format, and from that you get HTML, EPUB, PDF, slideshows, and man pages for free.[0] For math you get MathML, ASCIIMath, and LaTeX (along with a number of ways to render them.) It has a super nice syntax, is equally good at little docs and huge books, and you could theoretically write a proper academic paper in it with the LaTeX backend. And you always know what's going to happen when you try to mix bold and italic...

Also endorsed by Linus.[2]

    [0]: http://www.methods.co.nz/asciidoc/#_overview_and_examples
    [1]: Well, five minutes to be able to do everything Markdown can do; everything else will take a bit longer
    [2]: https://plus.google.com/+LinusTorvalds/posts/X2XVf9Q7MfV (comments)

Thanks, this is exactly what I was looking for. A simple syntax for documentation that is extensible with macros. I am glad AsciiDoc includes them, they are very useful.

I love AsciiBook. Everything but the name. If we ever get an amazing collaborative development tool like Github that has built-in AsciiBook rendering, this project would have been called "Scholarly AsciiBook (but we got Unicode support too I swear really)".

Edit: I was wrong about Github not supporting AsciiDoc

Github does render asciidoc using asciidoctor. What is missing in that that is possible in Markdown?

Well, I did not realize Github has had AsciiDoctor integration for two years now (I've been doing most of my scientific work in GitLab during that time). I retract that comment and regret the error. Hopefully they add it to Github pages some time soon.

To be honest, there was never really any conscious decision to pick Markdown over AsciiDoc. It was just serendipity that when I was looking to convert a bunch of my existing Markdown notes to papers, John happened to add (at Martin Fenner's urging) pretty robust support for citations to Pandoc. Being able to use an existing .bst files and Natbib author/year syntax with existing Bibtex database files, while also using CSL to format to everything else was pretty key. The syntax only worked in Markdown at the time, so I went with that. This project just simply grew into an experiment on how to get the least hack-y looking source document for what I needed.

Basically, I had a bunch of existing Markdown stuff typed with both my phone and my computer using NValt (already using the double-backtick math syntax as a hack), and I was looking for a way to reuse all that. If I had to do conversions or had to start from scratch, I would have just started another LaTeX Document, and then this project wouldn't be here.

As an aside aside: I personally think that one shortcoming of AsciiDoc is that Markdown ended up having a more mature application ecosystem outside of the most general purpose text-editing tools. I'm thinking of something like Scrivener/Ulysses which is really great for project writing and has robust markdown support while writing. I've also recently found out that TeXpad "mysteriously" formats Markdown syntax in the editor, and builds an in-editor TOC of it. This is in addition to with rendering support in basically every note-taking tool. Sadly AsciiDoc doesn't seem to have experienced a similar growth spurt in terms of popularity amongst developers, and as a result have a bit more friction to use.

Yeah. Unfortunately asciidoc isn't as popular as markdown and the ecosystem is small. I never paid much attention to asciidoc until I read this blog post - https://medium.com/@chacon/living-the-future-of-technical-wr... and realized that I have experienced many of the same frustrations with markdown. I tried out asciidoc and became a convert :)

For live preview of asciidoc there are some options - http://asciidoctor.org/docs/editing-asciidoc-with-live-previ.... Among them Atom with live preview works really well.

Just wanted to mention that GitLab supports asciidoc https://github.com/gitlabhq/gitlabhq/pull/7569

Thanks for the heads-up. I'm going to bug our admin again to upgrade.

Great! :)

fyi asciidoctor allows using markdown syntax inside your asciidoc file.

citation: http://asciidoctor.org/docs/asciidoc-syntax-quick-reference/...

In addition to supporting previews of AsciiDoc files, you’ll find that GitHub also does the “rendered prose diff” trick for commits and pull requests that include changes to AsciiDoc files.

AsciiDoc is a nice project, but I think that pandoc's variant of Markdown has a lot of advantages over AsciiDoc for academic writing. Let's just compare support for math and citations, for example. Pandoc's citation support is output-format independent: you can write the citations, specify a CSL stylesheet and bibliography, and you'll get the same output in every format pandoc supports. In AsciiDoc, as I understand it, if you want automatic citation support you need to use LaTeX, and then you're limited to output in LaTeX and PDF. Similarly for math: pandoc actually converts your LaTeX math to MathML (for formats that like that) or to native Word equations (for docx); in AsciiDoc, as I understand it, your LaTeX math will work if you target LaTeX, but if you want MathML you need to put MathML in the source. If you want to target both, then maybe you need two different source documents?

I don't have much experience with AsciiDoc, but I've encountered other limitations in writing pandoc's AsciiDoc renderer. For example, I didn't see any way to include multiple paragraphs (or other block content) in an AsciiDoc footnote. That's a deal-breaker for many academics.

I also want to emphasize something that is often not mentioned in comparisons of Markdown and AsciiDoc. As John Gruber emphasizes in his Markdown documentation, Markdown emphasizes ease of reading in source format. AsciiDoc has different priorities, and it sacrifices the readability of the source document to get them. Here's an example involving nested lists, from the AsciiDoc manual:

    1. List item one.
    +
    List item one continued with a second paragraph followed by an
    Indented block.
    +
    .................
    $ ls *.sh
    $ mv *.sh ~/tmp
    .................
    +
    List item continued with a third paragraph.
    
    2. List item two continued with an open block.
    +
    --
    This paragraph is part of the preceding list item.
    
    a. This list is nested and does not require explicit item continuation.
    +
    This paragraph is part of the preceding list item.
    
    b. List item b.
    
    This paragraph belongs to item two of the outer list.
    --

Markdown equivalent:

    1.  List item one.
    
        List item one continued with a second paragraph followed by an
        Indented block.
    
            $ ls *.sh
            $ mv *.sh ~/tmp
    
        List item continued with a third paragraph.
    
    2.  List item two continued with an open block.
    
        This paragraph is part of the preceding list item.
    
        1.  This list is nested and does not require explicit item continuation.
    
            This paragraph is part of the preceding list item.
    
        2. List item b.
    
        This paragraph belongs to item two of the outer list.

Note that the AsciiDoc is at least as easy to write, perhaps easier, because you don't need to worry about indentation. But the Markdown source is more readable; the indentation makes clear the structure of the list in a way that mirrors how it would be displayed in a browser or on the page.

The CommonMark is definitely more readable, but Markdown is just so sparse by design that every time I tried to use it for anything more serious than e.g. a blog post I found myself banging my head against the wall, so it's a tradeoff I'm happy to make - and we're talking about languages whose purpose is to produce readable output, so while a readable source file is definitely a nice thing I think it's hard to make a case for valuing that over functionality. To title a paragraph in markdown you can choose from an <h[1-6]> or a <strong>, that's it, and it's so tightly coupled to HTML (it's more of an HTML generation format like HAML than a document format really) that if you need to do anything remotely complex you're forced to either inline the HTML or bind yourself forever to external tools - in which case the goal of universal readability is defeated anyway.

Markdown's syntax constrains how much it can ever do, whereas AsciiDoc is designed to be extensible, providing macros at the "language" level and underneath that there's the whole DocBook toolchain: it seems to me that the clever move is leverage all this, rather than create yet-another-Markdown (and it's not like the syntax tradeoff is at all dramatic.) I think the footnote issue you mention can be done in block macros, but I'm not sure.

I might be getting a bit utopian here, but what would be really great is if all these markup languages sought to have their canonical implementation in pandoc, which would allow for the standardisation of a set of pass-throughs/filters/annotations for things like equations and citations.

Thanks for pandoc, btw. Just out of interest, is there a reason you haven't attempted an AsciiDoc reader? (Or have you?) Assuming it'd be quite a bit more work than the others, what with macros and so forth.

See https://github.com/jgm/pandoc/issues/1456.

Markdown (even this Scholarly Markdown) is a non-starter for me. It doesn't seem to support individual equation numbers for aligned equations. Plus, Scholarly Markdown oddly decided to put them at the beginning of the line rather than after. Why? The equation is more important than the number so it should be first. Plus, it's nice to have the numbers all aligned to the right margin.

You stated about the footnote support being a deal-breaker for many academics, the poor equation support is a deal-breaker for me. Most of what I write is full of equations, especially aligned equations. It's necessary to reference a specific equation that might be on a different page (or far above it on an HTML page).

It looks like Asciidoc has different ways of handling LaTeX equations for non-LaTeX output. It can render a PNG, it can use a Javascript solution, or it can output MathML.

I really want to address the math support, because I work with a lot of mathematicians, and one of my main goals is to support most of the latex AMS math features we're used to (as much as MathJax can support outputting on the HTML side anyways), and of course that includes aligned equations. In fact one of the first things I did was to figure out how to support multiline equations.

Turns out, you just need to write consecutive expressions one after another. Put ampersands in all these consecutive expressions and ScholarlyMarkdown will turn it into one giant align environment. If there is no ampersands, it generates a gather environment instead.

I apologize for this not being in the documentation. After all, I estimate that it is only about 30% complete. It really is in a dismal state. I originally didn't expect much people to see the site until later this year when I plan to launch it. This will all be rectified eventually, promise!

Here is literally a large align-block example straight out of the thesis that I'm currently busy working on (instead of the documentation). You can inspect the Scholdoc-rendered LaTeX code to see that you can really do align blocks reliably with this syntax.

http://goo.gl/9lErxa

If you output to HTML, it will just turn this same align block into a MathJax-friendly format, and hand it off to MathJax for rendering. This is what it will look like:

http://jsfiddle.net/hzvwa139/3/

Where the equation numbering is place is entirely a non issue. Why? It's not being decided by ScholarlyMarkdown, but by MathJax. Go to the jsFiddle example above and change the line that reads

  TagSide: "left",

To now read

  TagSide: "right",

and you get it now on the left. Scholdoc simply put those default setting there for convenience; you're not tied to it at all.

Why put the number on the left side by default? Think of what happens when you have a long equation, and when you're trying to read it on a narrow screen such as a phone. If the number was on the right, it may very well be cut-off by the screen, forcing you to scroll around to find equations. Sometimes layout decisions that made sense for paper for centuries doesn't make sense for screens.

I should point out that Scholdoc is only using MathJax on the HTML side for consistency; there is no technical reason why it can't use another renderer that understands LaTeX, but so far MathJax is the only one that even comes close to supporting all standard AMS features. Note that MathJax itself does HTML/CSS, MathML, SVG, and PNG (via a node server) output, and is entirely user-configurable.

I still think equation numbers on the left is a mistake. Ideally, for smaller screens it would split across multiple lines. Or, it could choose the positioning based on screen size.

For me, MathJax can't render my equations. I use some techniques to get a vector with an arrow on the bottom. Two commands I used all throughout my thesis are:

  \def \rV#1{\hbox{$#1$\kern-0.38em\lower0.85em\hbox{$\vec{}\,$}}\,}

and

  \def \Vec#1{\hbox{$#1$\kern-1.0em\lower0.4em\hbox{${\scriptstyle \rightarrow }$}}}

These won't work in MathJax (understandably).

> Ideally, for smaller screens it would split across multiple lines.

C'mon, nobody deserves to be subject to automatic TeX expression breaking. Even careful hand-tuning with help from the breqn package can often look pretty bad.

> Two commands I used all throughout my thesis are:

At the risk of turning this into TeX StackExchange, I just want to point out that these kerning manipulations are entirely possible in MathJax. The problem is that, maddeningly, they chose to have non-math commands like kern and lower work exclusively in math mode, and not outside one like in a hbox. Therefore, if you change your second command to something like this:

  $a \kern-1em\lower0.4em\hbox{$\scriptstyle \rightarrow$}$

then it will work in MathJax. Of course, this would then break in regular TeX because you're not supposed to use kern and lower in math mode. I keep myself sober by remaining optimistic about a potential overhaul in the MathJax latex processing code.

In the meantime, I think we would benefit from a stopgap service that renders these things in an actual TeX engine running on a server into high-res PNG (easy) or SVG (doubtful). There's been quite a resurgence in effort to make latex rendering into a scaleable web service lately. I'd love to collaborate with someone from, say, ShareLaTeX on that, if possible.

That's awfully annoying of MathJax because you need two separate versions of the LaTeX, one for MathJax and another for LaTeX output.

I know that there's dvisvgm for SVG output from DVI. I think there's also a tex2svg. Also, there's LatexRender-ng. So, there's a number of tools for SVG output.

I have yet to see a good stylesheet for AsciiDoc's DocBook output for creating a decent PDF. Do you have recommendations? I love AsciiDoc, but am only using it to generate HTML right now.

As @jauco said try Asciidoctor - http://asciidoctor.org/

In addition to default asciidoctor style, there are other themes available - http://themes.asciidoctor.org/preview/ (See bottom right for theme switcher).

You can also take a look at Pro Git book styles - http://git-scm.com/book/en/v2 which was written in asciidoc - https://github.com/progit/progit2

Thank you so much for the pointer to the git-scm-book (in the context of asciidoc/pdf output etc). Sadly the pdf-version, isn't exactly well laid out. It suffers from similar issues that a lot of html-to-pdf-based tools (although I assume it's sgml-to-pdf in this case?) -- horrible breaks, and it also "feels" wrong wrt. some spacing/etc. Generally standard LaTeX will look (much) better than this without tweaking, IMNHO.

On the other hand, they also have an epub style sheet, and it appears (a little oddly) that the layout of the epub is better than the pdf.

FWIW most "heavily optimized" custom LaTeX styles I've come across tend to feel like being slapped in the face with MS Word -- and I think I've yet to encounter any that actually improve on the "standard" styles in any meaningful way (with possible exception of the APA style, which is ugly, but as it has to conform to APA, it's ugly by design. And looks better than most other APA conforming styles I've seen).

Still, having a starting point makes the job much easier -- so this is a great resource.

My supervisor has been writing a book (for many years now) and he has a heavily customized LaTeX style that actually works quite well for his book. He has a lot of special needs for formatting so none of the default styles really worked for him. It's actually a pretty well done dynamics textbook. I've been looking forward to the actual final published text.

You should ask him if he's considered publishing the styles under a Free/open license. Even if parts of it is very specialized, I'd surprised if it couldn't work as an interesting starting point for other projects.

Next time I talk to him, I'll see about it. I don't talk to him that often now that I've finished my PhD.

Unfortunately not, XSL isn't really my thing. I'd probably consider trying to go via LaTeX, but that's because I like the way LaTeX looks out of the box and probably isn't appropriate for most things.

Try asciidoctor and its stylesheets.

Are there any good wiki softwares that use Asciidoc?

The github wiki software supports asciidoc and many other formats - https://github.com/gollum/gollum

Any others?

John MacFarlane (Pandoc’s author) is heavily committed to the Common Mark standardization effort of Markdown (in fact he’s the principal designer of the Common Mark spec).

Both Common Mark and Pandoc serve different purposes: the first is an initiative to counter fragmentation/balkanization of the Markdown ecosystem and in being so has to reckon with backward compatibility, consensus and adoption. The latter is a document conversion library, which, by design, needs to reckon with interchangeability between formats and may hence be hampered by the lowest common denominator as regards feature support. Internally, Pandoc keeps an Abstract Syntax Tree (accessible in json format), and defaults on its own flavour of Pandoc Markdown (featuring a Markdown superset of content element types).

Both the Common Mark community and John MacFarlane have made it clear their first and foremost focus is on standardization, not so much on extending the feature set. Yet, scholars and technical writers are in dire need for something more heavy-weight than the rather small set of features offered by Common Mark implementations or Pandoc Markdown. Hence the Scholarly Markdown initiative and the scholdoc reference implementation (Pandoc fork).

More on how Scholarly Markdown came about, can be read on the blog of one of it’s pacemakers, Martin Fenner; e.g. http://blog.martinfenner.org/2013/11/17/the-grammar-of-schol..., http://blogs.plos.org/mfenner/2012/12/18/additional-markdown...

Although I'm involved in both these enterprises, let's not confuse their goals.

CommonMark is currently focused on the task of giving a decent spec for the core syntax and robust, efficient implementations; extensions will wait til that project is done, but certainly aren't ruled out.

Pandoc has always been in the game of extending the feature set. Here are just some of the Markdown extensions pandoc supports: LaTeX math (which can be rendered in a variety of formats, including native Word and MathML), LaTeX macros, inline LaTeX, automatically numbered examples and cross-references to these, automatically generated citations (using CSL styles), super and subscripts, strikeout, figures, YAML metadata, definition lists, several styles of tables, fenced code blocks with syntax highlighting, header identifiers, and footnotes. scholdoc just adds a few things on top of all this (and many of them could be implemented in pandoc filters). As noted in one of the other comments on this thread, most of the features scholdoc adds are under active discussion in pandoc as well. So it's not that pandoc and scholdoc have different aims; pandoc just moves more slowly, because it has to worry about how features are implemented in many more output formats, and it operates under some other constraints that scholdoc rejects (e.g. trying to avoid the use of English words like "Figure" for syntax cues).

CommonMark should be mentioned on the ScholarlyMarkdown frontpage. I thought it was some competing project.

Don't they already use LaTex?

I'm still skimming the documentation so I apologize if the answer is obvious, but does anyone know offhand why this is implemented as a fork of Pandoc? Pandoc already has extended markdown features and the creator of pandoc is very much an academic (http://johnmacfarlane.net/), so is there are a reason why these contributions aren't part of pandoc proper?

Note: I'm the maintainer of this project

Basically this is forked because I've been wanting to see how much change to the AST is needed to include most of the academic-specific features. Since Pandoc's AST definition is a separate package from Pandoc itself and could possibly be a dependency of other projects, I thought it would be best to figure most of it out first and end up with just one proposal. Scholdoc has a much more limited number of input/output syntax, so it has much more flexibility when it comes to adding new document element types.

Consider this a self-motivated skunkworks project for Pandoc.

This is a fork because making these changes to pandoc itself needs a lot of consideration.

Internal referencing and attributes on figures are two things that are currently being discussed for pandoc. The discussion has been going on for quite a while though - hence people making forks.

Discussion on internal referencing: https://github.com/jgm/pandoc/issues/813

Discussion on image attributes: https://github.com/jgm/pandoc/issues/261

Thank you for the information.

This raises a couple more questions for me.

First, when searching I was able to find Martin Fenner's very interesting blog posts about ideas for a "Scholarly Markdown" and, as those issues and the first link on the scholarlymarkdown.com site reference, he appears to be associated with a separate "scholmd" project, also called "Scholarly Markdown," which is apparently a related project that itself is a fork of the Python markdown science project:

scholmd:

http://scholmd.org/

https://github.com/scholmd

Markdown Science:

https://github.com/karthik/markdown_science

However, it's unclear what all of the relationships are between all of these projects and forks.

Secondly, since some (or all?) of the changes are being discussed in the Pandoc issue tracker, are these changes intended to be submitted to Pandoc in pull requests? I don't currently see any.

First: I'm not sure of the exact origins of things. The way I see it, academic markdown is more of an ecosystem of tools with a lot of overlap. There is no one single markdown workflow right now when you want to do do academic writing. I think this is because no one is sure what the final spec should look like and people are trying things out and seeing what sticks. I feel that there has been some convergence in the last 2/3 years though.

Second: The PR for image attributes is here: https://github.com/jgm/pandoc/pull/1806

There isn't a PR for internal referencing yet because the implementation hasn't been worked out yet and it isn't a simple change (should there be a native representation, should it be a filter, which syntax should we use, what about the existing citation syntax...).

Note: I'm the maintainer of this project

The series of blog posts by Martin (and his efforts with John in getting citations to work in Pandoc) was the impetus of this project. I've reached out to Martin several months ago for comments, but I've not heard from him since. I guess he's very busy with his day job at PLOS. If he's willing, I'd very much like to reconcile this project with his efforts. The goal is, after all, better authoring workflows for all academics compared to the status quo, and it's going to take some concerted effort to get us all out of this giant energy well we got going for a few decades now.

Scholarly Markdown is very much a group of like-minded people, and we had a workshop with lots of good discussions in June 2013 (http://blog.martinfenner.org/2013/06/17/what-is-scholarly-ma...). What it has not been until the recent effort by timtylin is a specific set of tools, or spec.

Everyone seems to have an opinion on how to do this right, and that is part of the reason why the whole concept is pretty fragmented. Some of my thoughts:

Pandoc is the markdown converter that comes closest to what most people need, so I am happy to stick with it. I personally don't think that a fork is viable, things are already hard enough as it is.

Scholarly markdown is a solution for 80% of use cases, people writing math-heavy texts are probably better of sticking with Latex.

Scholarly markdown needs to be a community effort, I don't see any other way on how this can succeed

Hi Martin! Thanks for dropping by.

> I personally don't think that a fork is viable, things are already hard enough as it is.

I don't think so either. Scholdoc as a fork was always intended to be a stop-gap measure to quickly test out ideas. Pandoc's use of relatively standard Parsec is easier to hack, and lots of other subsystems like citeproc remain crucial. Scholdoc changes Pandoc's AST, so any discussion of re-integration is going to be a non-starter until at least 2.0

For this kind of workflow to be viable, 95% of the required effort is not going to be on the syntax/converter anyways. The real hard work is still ahead.

> Scholarly markdown is a solution for 80% of use cases, people writing math-heavy texts are probably better of sticking with Latex.

I agree, except I also think that there can be a 80% situation for math. I work with a lot of applied mathematicians/electrical engineers, and the math system in Scholdoc is designed with them in mind.

I really think that the ultimate goal is to arrive at many good ways (of which this may be one) to produce semantically-relavant open interchange format such as JATS. I assume this is what PLOS is trying to achieve as well? I do know that several people at PLOS is vehemently opposed to Markdown and what it stands for.

> Scholarly markdown needs to be a community effort, I don't see any other way on how this can succeed

Definitely. The best we can hope for is to occasionally stir this pot once in a while and hopefully something will spontaneously nucleate once the time is right.

This was my first thought as well. Pandoc has a rather excellent infrastructure in place to let you extend and build on it, so if you instead choose to fork, there better be a good reason...

Pandoc filters let you do a lot of things provided you stick to the defined types. If you want to add new types of element, like this does, then the only practical way is a fork.

(except for really simple cases)

However, in most cases you could just RawBlock (or was it BlockQuote?) with specific attributes, and then intercept those in a filter and convert to Latex.

This was basically my original approach. However, after working out the math syntax I realized that some things, like the double-backtick inline math, just can't be accomplished without a pre-filter. At that point I decided to just start playing with the parser code.

I also became super-convinced that some level of AST change was necessary to keep things sane, and since I wasn't able to use the existing Math and Image types anyways (they're not attributed), I ultimately just started a new AST type package namespace called "Scholdoc". Everything just evolved from there.

Yes, BlockQuote can be used as a generic block container. It doesn't have attributes itself but you can put them on a header at the start. For generic inline elements you can use Code.

Markdown syntax for generic containers (Div and Span) hasn't been implemented yet but the discussion is fairly mature now:

https://github.com/jgm/pandoc/issues/168

https://github.com/jgm/pandoc/pull/1791

A similar effort of introducing math/figures/refs into markdown is the `softcover markdown` syntax. It's basically markdown, with latex commands allowed:

https://raw.githubusercontent.com/softcover/softcover_book/m...

The beautiful idea in markdown is that it allows you to mix (non-container) HTML tags in with the .md and it just works. Softcover markdown is in the same vein, allowing the more readable markdown for main copy, and intermix LaTeX tags as required. Beautiful if you ask me. Or at least beautifuler than ```math ... ```math.

The "backward compatibility" of ScholarlyMarkdown with basic markdown is a cool feature as many tools/plarforms exist that "support" .md now, but to preview you'll still need something that renders the equations, so strictly speaking ScholarlyMarkdown is a new markup langauge.

It seems that Markdown is starting to get a lot of disgregated reference implementations. Some months ago some important Markdown users were behind CommonMark (http://commonmark.org/). I'd think that it would be best to join this conversation rather than splintering Markdown even more.

I'm in fundamental disagreement with the principle behind CommonMark. They prioritize standardization over practical usability, and for that reason people will keep splintering markdown.

For example, they do not plan to add syntax highlighting blocks (the ```some code``` on GitHub) to their implementation, because they believe that it is outside the scope of markdown. Then, because a lot of people actually need this feature, they still have to patch or extend or plug-in the functionality into any implementation of CommonMark, leading to fragmentation again. And frankly, the reason that I start writing more markdown is precisely because of the syntax highlighting ability. Oh, and to write scientific articles, math formulas are a deal-breaker.

They want to have a standard, unambiguous syntax specification, a suite of comprehensive tests and a cleanly implemented parser. They want to unify the community of markdown users and developers. All of those are commendable goals. But at the end of the day, it doesn't satisfy my needs, so I'd rather use a messy, poorly specified markdown flavor or even just render the markdown with GitHub's service.

"For example, they do not plan to add syntax highlighting blocks (the ```some code``` on GitHub) to their implementation, because they believe that it is outside the scope of markdown."

To correct the record, fenced code blocks have been there from the beginning: http://spec.commonmark.org/0.18/#fenced-code-blocks.

I guess many has changed since I last looked at CommonMark and perhaps the developers are pragmatic after all. I stand corrected and will try out their implementations now.

They should concentrate on a wide variety of features and make it modular. That way, I could download a build with bold, italic, and spoiler syntax. Or, someone else could download one with only lists, strikeouts, headings, and math formulas. When you're missing even one feature, people go elsewhere.

Lastly, why is everyone hellbent on following the original syntax like it's the word of God? Anytime I hear someone complain about an issue in Markdown, people point to the 2004 syntax and how it's true to spec. Users don't care, they want something usable. I'd be happy if someone creates a completely new spec, and I never hear the word Markdown again. I'm tired of the same issues being passed down.

An alternative would be like what `LaTex` does, a `\usepackage` or similar directive that declares in the source what extension this document requires. Maybe a central repository like CTAN as well.

Wow, I've never seen CommonMark -- hope that comes into common use over everything else!

Why? This really feels like cramming a square peg into a round hole. Anytime anyone wants something more complex than simply lightly formatted text they will have to use other tool (LaTeX), so why not just use LaTeX in the first place?

The way I see it, the main feature is being able to plug math formulas in, rather than formatting documents. It allows you to focus on your content (one feature of Markdown) while still being able to write mathematics.

"It allows you to focus on your content (one feature of Markdown) while still being able to write mathematics."

That's what LaTeX does too. Its markup, generally, is semantic.

In my experience LaTeX is quite poor at separating content from presentation, at least when you're unwilling to rely on the default styles. All the layout commands are context-sensitive, so I often end up writing presentation code in the middle of my content, or accidentally breaking the layout by moving content around.

You should be putting your alternate style in a .cls file and not inline. If you are super picky about format, then none of these tools are for you.

Alternatively you can use org-mode and extend the document with latex where appropriate.

This thing is certainly needed (vide this discussion: https://hackpad.com/New-scientific-markup-language-utAjFcYuv...), but what would be really convincing is examples (or I am missing them?).

As a side note, a lot of Markdown + LaTeX + Code can be done in IPython Notebook. (Though, there are some things absent, like referencing citations or other equations).

I read your link and, per my main comment, I'm pretty sure AsciiDoc answers 99% of your needs.

http://www.methods.co.nz/asciidoc/#_overview_and_examples

Cool! Any examples of using AsciiDoc for math or scientific notes? (I.e. with formulae, references...)

Well here's some formulae:

    http://www.noteshare.io/section/the-fundamental-class-of-projective-space
    http://asciidoctor.org/docs/user-manual/#using-multiple-stem-interpreters
    http://www.methods.co.nz/asciidoc/latex-filter.html
    http://dblatex.sourceforge.net/example/dblatex/example_mathml.pdf

As for references, it has it's own lightweight bibliography system out of the box, but there's a plugin[0] for BibTeX too, and DocBook has full-on support for BibTeX so it's just a matter of tooling. AsciiDoc gives you DocBook, and DocBook gives you pretty much everything.[1] The whole thing is completely extensible at multiple levels (macros, XSL stylesheets), so adding any essential features it doesn't already have is certainly much simpler than starting from scratch!

    [0]: https://github.com/petercrlane/asciidoc-bib
    [1]: http://pub.hdcrd.com/kb/Dev/Documention/LaTeX/Tool/Dblatex%20%28DocBook%20to%20LaTeX%20Publishing%29/0.3/manual.pdf

I've just updated the website to include a link to a repo where I am putting all the examples I have permission to share.

https://github.com/timtylin/scholdoc-examples

More will be added as I tweak the syntax. My hidden goal is to add enough to ScholarlyMarkdown to be able to reproduce around 70% of ArXiv.

Yes, but it's not ideal. Since everything is done through Javascript, we have to first identify all the math, then hide it from the Markdown parser, then parse the Markdown, then restore the math, then call MathJax on the now-HTML. Equation references/numbering exist, but are turned off by default. It's a minor change to your MathJax configuration to enable.

I created a similar thing for my personal website:

http://www.dllu.net/programming/dllup/

which handles math using svgtex instead of clientside MathJax for faster rendering. It also compiles to both html5 and LaTeX. However, dllup is overall less polished, missing some features (labelling equations, sections).

Interesting to see this come up. I've been migrating a couple projects to babel, the emacs org-mode literate programming style. Starting with the hello-world of babel, my emacs config file.

Babel and Org have clunky syntax but the mode takes care of that. The combination isn't perfect but it's very powerful, in particular the ability to chain several languages together in flexible ways.

Coming up with more ways to combine pretty formatting with syntax highlighting is polishing a pretty smooth surface. I'm dreaming of a really slick syntax and operating environment, like babel if it wasn't tacked onto org. In the meantime, using what we have.

Alot of interesting work, judging from the site, but this still feels like reinventing the wheel to me. In addition to AsciiDoc, there's MultiMarkdown (http://fletcherpenney.net/multimarkdown/features/ feature list is nearly identical), and probably another 3-4 that I haven't come across.

Of course, having spent the past few months writing an app that basically reinvents this same wheel (http://www.eqeditor.com/writer/) I suppose I'm in good company.

Another thing I'm curious about is how the ScholarlyMarkdown authors would go about trying to get papers formatted in this way to go into a database like PubMed but that's pretty far down the track, and not addressed by any of the other solutions either.

The goal is eventually to be able to, like what Authorea is trying to do now, just "slap on" various LaTeX style templates during final rendering (similar to how static blog engines format plain md posts using templates). It's going to be PAINFUL but I think it is eventually doable with enough elbow grease. Markdown is a nice starting point because the features are so spartan that you actually have some hope of finding the "greatest common denominator" document model that can pretty much map injectively to all the major journal-specific LaTeX document-classes.

I'm imagining some kind of online component like ShareLaTeX that will become a clearinghouse for a number of tested and proven conversion paths, and can handle compiling a ScholarlyMarkdown document to different formats. This project won't go anywhere if this can't at least be done for major houses like Elsevier, PNAS, Phys Rev, etc.

If you want to write scientific documents with markdown, beware that pandoc already supports inline TeX (math with $$ and TeX commands starting with \) when doing md -> latex conversion. That has been my setup for quite some time.

If you want me (as an academic) to use ScholarlyMarkdown, then you can't force my collaborators to use ScholarlyMarkdown as a consequence.

There should be a nice ScholarlyMarkdown -> LaTeX cross compiler for starting simple documents in TeX and then sharing with collaborators (pick some obvious defaults or allow a config file to get fancy). But more importantly, if I am joining a project that already has a bunch of LaTeX wizardry going on, I should be able to seamlessly and implicitly edit the text parts in ScholarlyMarkdown without my collaborators knowing.

Can ScholarlyMarkdown do this? If not then I'm not really interested.

I've actually been wondering if this is possible from a theoretical standpoint. I'm thinking you can use Pandoc's LaTeX to MD conversion mode, save changes to a copy, wdiff to get the total change set, then somehow convert that into a LaTeX patch. Whether you can guarantee that this won't break anything is going to be a really challenging problem, as my head is already swimming with edge-cases. I guess we'll know the day that some genius comes up with automatically generated non format-breaking Critic Markup diffs. At least the problem should be easier than LaTeX --- look how long it took for the latexdiff tool to be what it is today, and it still breaks often when you have anything slightly exotic.

I'm currently using a workflow in my thesis where I use ScholarlyMarkdown to write individual chapters for final inclusion into an existing LaTeX book document. I find that ScholarlyMarkdown works quite well this way, and it potentially allows collaborators, since individual parts are isolated.

In any case, my point is plain and simple: I collaborate in almost everything I do, and my collaborators are my primary concern when I choose a tool. If they use notepad and Dropbox, I need to make sure my tools don't conflict with that. There's no way in hell I would ask a collaborator to learn to use git or a new markup language just to work with me.

I'd love a tool that works with this philosophy, and I feel certain anything like ScholarlyMarkdown won't catch on in my field (theoretical computer science) without such tools.

As a thought exercise, assuming for simplicity that you can isolate the parts you're authoring, so that your changes only involve additions to other people's work: what kind of additional metainfo would you need for this workflow? All defined labels and macros? Available bib entries? What else?

I think it depends on how I'm viewing the file. If I'm just editing in SM and building the tex as usual, then I don't need anything extra. If I'm trying to convert to all sorts of other documents (which I haven't ever really done) then labels, theorem environments, one-line macros, and bib entries cover almost every tex-specific thing I use.

Here is an excerpt from a typical paper's macro section [1]. As you can see they're mostly one-liners to remove the need to keep typing textup and mathbb, simple mathoperator definitions and such.

[1]: http://pastebin.com/RL1gejEZ

I think a good way to build this into SM is to run an existing document thorough latex and looking at the aux file. This approach would be faster and almost certainly more robust, similar to how most latex build scripts look at the fls recorder file for the list of external includes.

The only only thing you can't get will be the user-defined macros (and of course bib entries that doesn't already exist), but there is already a consistent mechanism to define your own macros in SM via the "math_def" block. They "do the right thing" in the sense that if you render to latex snippets then it wouldn't redundantly include these macros in the output.

That's probably never going to happen (for any Markdown-based project).

As you probably know, TeX is a full programming language (albeit an odd one), which means that to compile (La)TeX to Markdown, Markdown needs to support all the features of TeX, which means that it has to be Turing-complete. Not to mention the fact that parsing TeX is kind of nuts too.

There's no need to convert all TeX back to markdown; the converter only needs to recognise the sub-set of TeX that it produces.

If anyone edits the TeX to fall outside that sub-set, it's reasonable to just embed those parts verbatim in the resulting Markdown.

Exactly. And I have a very strong feeling that without the ability to collaborate implicitly (at least on the text/math mode part), no replacement for LaTeX will catch on in my community (mathematics).

My guess is that he is building the PDFs through latex/xetex.

I built something similar to that, but instead of forking Pandoc (and investing the time to learn Haskell), I just added filters through Python. It's not at fast, but you can intercept mostly anything.

Thus, the flow ends up as:

paper.md -> Pandoc -> paper.json + metadata -> paper.tex -> paper.pdf

I want the flow to be

paper.tex -> paper.md + metadata ->(merge) paper.tex -> paper.pdf

I wonder how long it'll be before Gruber complains about the name.

Part of the beauty of markdown, imo, is its simplicity. If I ever have a question about markdown syntax, I can glance at the original docs and have the answer in less than 30 seconds.

Looks like an interesting project. I may be missing something, but I think in terms of math especially it would be hard to use Markdown instead of something more robust like LaTeX which supports proper mathematical notation.

Edit: yes, I was missing something. Scholdoc supports MathJax (which has browser, Office, and LaTeX support).

Small issue: The title isn't showing up in Firefox (or Chrome) because there are two title elements and the first one is blank.

I actually like the HTML/CSS solution too. http://thomaspark.me/2015/01/pubcss-formatting-academic-publ...

It's more difficult but I feel like there's a lot more potential.

Yep, that's the idea. Published work today should be able to take advantage of modern screens and its ability to resize, reflow, and animate.

This looks very interesting but I have a problem with this. I think MOST of these features could have been build just with clever uses of Pandoc filters. This means that i) it's fully compatible ii) it's very easy to extend

Why not asciidoc?

Around 2 years ago I wrote a patch for chjj/marked that supports MathJax. Despite heated discussion, author never noticed and it wasn't merged. Just a glance at how fragmented this markdown world has become.

I'm annoyed that there's still no way to underline in markdown.

My spontaneous reaction is: Why don't use the existing MediaWiki syntax? It has gone through the test of years (decades?) of real-life needs, has support for (latex) formulae, everything you might need around images and references etc, one of the most powerful template systems I've ever seen, and the list goes on and on.

Additionally there is a very stable, performant and flexible reference implementation implemented as a web app (mediawiki) with excellent import/export in XML format (and the list goes on ...), versioning with syntax-highlighted diff, etc etc etc.

What have I missed? :)

MediaWiki syntax is incredibly grody ('''bold''' / ''italic'', tables, template logic, etc), and has exactly one implementation in common usage (MediaWiki itself -- written in PHP, and horribly convoluted), and has no formal specification. The diff feature you're referencing as a "syntax-highlighted diff" is a simple diff of the source -- it's entirely ignorant of formatting, and the XML export is similarly just an export of the raw source.

Nobody likes MediaWiki syntax, not even its own users. It's awful. The only reason it's still used at Wikipedia is because there's too much content to reasonably convert.

> It is a fork of Pandoc and is build upon the same parsing engine.

=> built

Reply on news.ycombinator.com