Hacker News Reader

Hacker News

Comment by jak2k | Hacker News Reader

Parent Full thread

jak2k

Why even convert HTML to markdown? Isn't it usually the other way around?

View on HN

1. For reading. I simplify all documents to mdown then render back to html on my readers...e-readers and rss

2. Saves context for llms and they are often trained on markdown so work best with it.

3. For search. Can search markdown much better than html with postgres

Wrote https://markdown.download to help me with these

We have HTML templates for sending transactional email in our SaaS applications.

The templates are very basic, eg. <p> <b> <a>, etc. Users can also customise these via a WYSIWYG editor.

We then use turndown.js to convert the rendered HTML email to markdown which we then use for the text version of the email.

Storage.

My use case involves scraping job boards so that I don't have to doomscroll them myself anymore, and storing them in Markdown makes them smaller while also removing a bunch of extraneous classes and structure.

Further, the side project I'm working on for managing all of this can then render them in a way that makes sense.

you might be interested in https://www.kadoa.com/use-cases/jobs if you prefer to fully automate the process of job boards scraping

Only use case I can think of is to save web pages to your MarkDown notes. Web links usually break after a year or two. Unfortunately.

I created an RSS reader which has a uniform reader mode. I use something similar to this to parse each RSS article to a similar format. I'm sure there are many other use cases also.

I had to do this to recover my personal blog after both it and the backups had been lost due to two unrelated snafus during covid. I downloaded the pages from the internet archive and used my own shellscript to extract the text as markdown and then republished it using a static site generator.

Not exactly a common usecase I wouldn't think but it's good to be able to do this.

Advent of Code exercises are almost pure markdown, but rendered to HTML.

I’ve sometimes been converting it back to md to include the text for each exercise alongside my solutions.

In my case I used a custom HTML to Markdown converter that was specifically built to support only what I needed in order to convert those Advent of Code exercises to markdown.

Mine was also written in Rust.

If you want to store markdown in your database but you want to user to use a basic wysiwig/content editable editor it can allow you to not go through the full blown markdown editor.

Similarly for sanitization, if you want to really reduce the subset of allowed tags, and to normalize input it's a pretty good intermediate format. Did this for a classified ads site, that included listings from external companies... it was easy enough to shift H1-H3 to H4-H6, keep basic formatting elements and eliminate the rest.

Markdown in the database is also easier to look at, reason with and takes up a lot less space. Especially if the content was pasted from say MS-Word to a Content-Editable field... omg the level of chaos there.

To feed content to LLMs

"But it's not structured!"

I swim around a lot in the "XML High Priesthood" pool, and the latest new thing is this: AI (sucking down unstructured documents) isn't capable of efficient functioning without Knowledge Graph, and donchaknow a complex XML schema and a knowledge graph are practically the same thing.

So they're glueing on some new functionality to try and get writer teams to take the plunge and - same old same old - buy multimillion dollar tools to make PDFs with. One sign of a terminal bagholder is seeing the same tech come up every few years with the latest fashionable thing stapled on its face. They went through a "blockchain" phase too, where all the individual document elements would be addressable "through the chain".

Anyway . . .

Anyway, thing is, there's a teensy shred of truth in what they're saying, but everything else about what they're suggesting would, I think, either not work at all, or make retrieval even less dependable. Also, to do what they're trying to do, you don't actually need a gigantic full on XML schema. Using Asciidoc roles consistently would get you the same benefit, and would save a hell of a lot of space in a very limited window.

Yeah, this. Markdown uses less tokens than HTML and most LLMs have been trend on large amounts of Markdown.

That's why tools like this exist: https://jina.ai/reader/

Demo: https://r.jina.ai/https://news.ycombinator.com/item?id=40695...

Additionally, when you have strict input token limits: it’s way easier to chunk Markdown while keeping track of context than it is to chunk HTML at all.

fun project, saving pages to Obsidian

Obsidian supports html->md natively. Just copy the texts in browser and paste them in Obsidian.

Reply on news.ycombinator.com