Comment by radhakrsna | Hacker News Reader

Comment by radhakrsna | Hacker News Reader

Hi everyone

I am delighted to share my new project "TLDR This" with you guys.

Problem : There's so much content out there but too little time to read. So many times, it happens that there's a long article that we're interested in, but we don't really feel like scouring through it to extract relevant information from it.

Hence, I created "TLDR This" to help you navigate relevant content quickly and easily, without having to read the whole thing.

Steps to get cracking - 1. Copy and paste either the URL or the text of the article you'd like summarized. 2. Press the "Process Text" button, and you are good to go.

TLDR This also comes with a chrome extension, allowing you to summarize any webpage at the click of a button.

How to use the chrome extension? Just click the “tl;dr” button in Chrome's toolbar on a webpage which you'd like summarized and within a few seconds, you'll get the < 5 sentence summary right there.

Please let me know if you have any feedback/suggestions.

Thanks a lot

Introduce this article: https://johnnyrodgers.is/burnout

Each section subtitle is displayed as a sentence. Could you maybe ML generate a 1 sentence summary for section? I don't think there is an ML which could extract the take aways from each section and make a summary correctly, but that was my original expectation when I saw it first time.

> I don't think there is an ML which could extract the take aways from each section

There's plenty of abstractive and extractive methods for text summarization. In fact, that's what I'm working on right now, is to develop a news article summarization system for longer articles.

If you're interested in learning more, there's an excellent survey paper on ArXiv: https://arxiv.org/abs/1812.02303

is there any working demo to play with? I am not into ML and won't read the whole paper... I expect these sort of mechanisms to make some weighting of the morphemes and sentences and replicate the top score ones whereas a human made summary is based on context and semantics given by humans (for example, the relevant part might be only relevant during the current zeitgeist or for some peoples bias)

I'm not aware of any working demos. In terms of the weighting, your intuition is correct, there's a way how a machine learning algorithm can learn which tokens (what you call morphemes) are of significance automatically. That's called an attention mechanism. It learns what tokens are important and then passes that into a "context vector" of sorts, which is then passed into the generator network (which is some form of recurrent network).

The disadvantage of using only token-based attention is that it's still not enough context for large spans of text (> 1000 tokens), and I'm actually having this issue now in my work. That's where things like applying attention to chunks of text is helpful.

I'm still trying to fully understand attention mechanisms, so I hope this comment made sense.

but facing such problem with an absolute lack of abstraction on the ideas behind obviously cannot work at any non straightforward example, isn't it? there must be some other steps or algorithms or additions to this process; the language is a protocol to encode a message, a representation about abstract concepts and ideas semantically referenced. It is not the same the HTTP server configuration values and statistical analysis of a xml text encoding those; and that still would be straightforward communication, not metaphoric speech or indirect references.

As long time language learner I can tell you that the most significative words in a text are the less common (that's why learning a language by lists of common words does not work out of the box). In the sort of process you mention the words with more meaning for the idea to communicate might have easily less weight than other more common tokens at the same texts, and it very easily depends on the communication style of the subject.

For me this way of making a summary, looking at the language as a sum of interconnected tokens, sounds like trying to replicate a paint by measuring brush strokes directions and pigmentation (vectors of values) rather than trying to understand the abstract concept or idea behind (a vase with flowers, fruits) and recreating it (use the skill of extracted with the vectors to recreate the abstract idea which was encoded behind).

I think I might end up reading the paper you mentioned.

To address how computers establish context, usually the tokens are sent through an embedding layer (basically turning a word into a dense vector). Word embeddings, surprisingly, reflect the real-world semantic relationships behind words (for instance, king - man + woman = queen). It also happens that the algorithms for creating word embeddings group words with similar meanings/semantics together (for example, king and queen would be closer together in n-dimensional space). Overall, embeddings allow a computer to look at a language as more than a bunch of interconnected tokens and actually incorporate the semantics of each word into its predictions. Of course, this isn't perfect, which is why we need attention mechanisms and various other methods to help a computer try to understand context long-term.

This is a very good article. Thanks for sharing!

Some info about it would be great? Do you use an AI model? Do you do semantic analysis? Is it open source?

I tried it on a wikipedia article and it didn't work too well.

> TLDR This also comes with a chrome extension

Firefox uses the exact same extension format, and except for non-trivial cases, “porting” an extension to Firefox involves literally no code-changes.

Consider publishing your extension for Firefox too. You just need a Mozilla-account to publish it on addons.mozilla.org. That’s pretty much it.

Hi,

That's a very good suggestion. I didn't know it was that easy to build an extension for Firefox. I will publish it on the Mozilla store as soon as possible.

Thank you very much for taking the time to use the app and providing your feedback.

Hi! We have added the link for Firefox extension. Please check it out and provide your feedback. Thank you!

Congrats! It worked well on a few sample articles that I tried. Can you provide background on how the text is summarized?

Thanks a lot. Sure, first we create a list of all the individual sentences in the article. Then, we use the TextRank algorithm to rank each of the text sentences. Finally, we select the top 5 most representative sentences from the article.

How do you create list of sentences from the article? How do you know where article starts and what is just navigation or fluff.

Firefox's reader's view is able to do that.

Check out their Github repo here - https://github.com/mozilla/readability

>You need at least one <p> tag around the text, you want to see in Reader View and at least 516 characters in 7 words inside the text.

Source - https://stackoverflow.com/questions/30661650/how-does-firefo...

Hi, We used a python library called newspaper to do it.

how does this compare to newspaper's built-in summary feature? What is different?

Unless you make this painfully transparent, a rational person would have to assume some intent of bias. No offense intended - I'm stoked you made this as its one of those tools I've always intended to make myself. But honestly, given the revelations of the last few years - unless we can verify as a community that there isn't bias, we have to assume there is.

Reply on news.ycombinator.com