2markdown – Transform Websites into Markdown | Hacker News Reader

2markdown – Transform Websites into Markdown

2markdown.com

17 pointssmartmic20 comments

Why not just use something like https://github.com/mozilla/readability

And not pay $0.01 per request?

There’s a node version too https://www.npmjs.com/package/@mozilla/readability

Not the OP, but I tested readability for this purpose. My use case was converting web pages of API documentation into markdown for AI coding.

Readability works pretty well on "newspaper article" or "blog post" type content. But fails badly on the more structured webpages that I was interested in like API documentation, etc.

I'll stick with using Markdownload (https://github.com/deathau/markdownload) when I need to do something like this.

I built a similar thing in python using Playwright and Pandoc [0]. It's used by aider's `/web <url>` command that lets you paste a markdown version of any webpage into your AI coding chat. This helps if you want to include docs for an obscure or non-public package/api/etc with the LLM while coding.

I really value dependencies which are easy for all users to install, cross-platform. Playwright is nice because it has a simple way to install its dependencies on most platforms. And the `pypandoc` module provides a seamless install of pandoc across platforms.

The result turns most web pages into nice markdown, without requiring users to solve some painful platform specific chromium dependency nightmare.

For example, it produced this markdown version [1] of the pyinfra getting started doc [2].

[0] https://github.com/paul-gauthier/aider/blob/main/aider/scrap...

[1] https://gist.github.com/paul-gauthier/6c8dd16a6bceb9fd287494...

[2] https://docs.pyinfra.com/en/2.x/getting-started.html

These two blob posts by the author seems to be revealing:

How to Build a Startup Without Funding https://andinfinity.eu/post/2024-04-02-how-to-build-a-startu...

What's the minimal set of user stories for a Micro-SaaS MVP? https://andinfinity.eu/post/2022-06-06-whats-part-of-an-mvp-...

Wow, "pay as you go html to markdown" seems like a very hard sell in this year of our lord 2024. I've got a brace of converters for various parts of this problem, some of them are pushing a decade in age. You have products like Zapier or the open source ActivePieces to restrain extraction to specific elements. Finally, several pre-wrapped AI models for this specific purpose.

I have been using lynx to integrate a chatbot with web content. Here's the Python function that converts a website to plain text:

  def website_to_text(url):
       output = subprocess.check_output(["lynx", "-dump", "-nolist", url], stderr=subprocess.PIPE, universal_newlines=True)
       return output.strip()

While a hosted service can be convenient, I find that using a text-based browser like Lynx gives almost the same results and consistent results.

I used to be intrested in converting/saving websites to my own archives. The web is rotting away and saving a local copy is the easiest way to make sure I keep my references.

This, this ain't it chief. If you're looking for local saving, use firefox extensions. MarkDownload and SinglePage are both very good for backing up single web pages.

pandoc as a service, mad times we live in, best of luck I suppose

Not interesting in the service but fascinated by this new wave of re-modularization and re-integration of services through the lens of Ben Thompson aggregation theory.

Modularizers like 2markdown are building single purpose services optimized for LLM workloads

Aggregators like ChatGPT fights for end-to-end AI experience.

It's just exciting to watch.

I have a wrapper for the many ways you might convert something to a text document in the terminal.

https://github.com/reagle/pandoc-wrappers/blob/main/doc2txt....

It might be helpful when accessing news websites outdoor (since my internet provider rate limits hard). It’s basically a proxy, right?

I’m not even going to try though because there’s a signup wall.

> From this

> This content displays when the browser does not support custom elements.

> To this

> This content displays when the browser does not support custom elements.

this happens with ublock origin on

> empower your AI models

pardon?

I suppose that AI models may take markdown as input?

But they can take html as input as well..

I saw that and immediately clicked off. I do sometimes want documents in markdown format (though Obsidian is pretty good at just converting whatever you paste in automatically), but it's clear this is a lazy $current_buzzword-hype cash grab. Not worth HN front page space.

no, its a real need. llms need to ingest websites and markdown is more efficient than html.

Should definitely use a try 1st one free without login demo (maybe with a strong captcha if missuse costs are an issue)

The markup on this seems insane. A Cloudflare worker could easily do this and you would make 10K on 30 cents.

If you’re looking for an open core version of this check out firecrawl.dev

Reply on news.ycombinator.com