Show HN: TLDR This – Auto summarize any article or webpage in a click
tldrthis.com
tldrthis.com
1. Split the text into words
2. Rank each word based on how many times it appears in the text. For example, a word that appears 10 times gets 10 points, and so on.
3. Rank sentences based on the sum of the scores of each word inside them.
4. Return the top N sentences by score (N is up to the user), in the order in which they appear in the text.
For extra fancyness, exclude the most common articles and prepositions and give 2 points to proper nouns.
Works surprisingly well.
The SMMRY page points out some extra NLP-related grunt work in addition to the high-level steps you list, like:
>Associate words with their grammatical counterparts. (e.g. "city" and "cities")
>Detect which periods represent the end of a sentence. (e.g "Mr." does not).
[1] https://www.reddit.com/r/autotldr/comments/31b9fm/faq_autotl...
The method you described is a part of our algorithms but more steps are needed to make it give meaningful results and make sure it works on different kinds of articles.
Text summarization works as a good toy problem, because it leads to two harder problems: 1. text extraction (how to distinguish content from non-content like ads) 2. q&a (given text and a question about the text, how can you produce an answer).
You can improve contextual summarization by splitting the x sentences into x/n buckets. Then based on the percent of article to be summarized (eg return 60% of the article), pick the sentences ranked in the top 60% of each bucket. Then do this for all the x sentences, ie top 60% across buckets, and combine them together.
This prevents the bias rising from picking a sentence with a lot of critical words.
... So, don't do what the late Thag Simmons did...
Maybe final paragraphs should be more highly weighted? That's often where the conclusion is.
Longer sentences had an inherent advantage, so I controlled for that by reducing sentence scores as a function of the sentence length.
1. From the articles I tried, the summaries seem to be very basic. They don't seem to capture the essential points of the articles it is trying to summarize.
2. I tried the 'advanced summarizer' too. Here again, it seems to have the same problem. Worse, it seems to skip parts of the article, especially if they are beyond a certain length.
3. The landing page is nice. But the product seems to be targeted towards people who want to share a summary of a random blog post rather than try and save time reading the article.
In my opinion, SMMRY, as mentioned in another comment and which I've used since whenever, seems like a much better product. Additionally, SMMRY also gives you the capability of expanding the number of lines of the summary in case you've found the article interesting and want to read a few more details of it, rather than the full thing.
SMMRY: https://smmry.com/
Of course on it's face it seems nice that it saves us time. But it's no secret that the reduction of complicated topics into simplified one-liners leads to less understanding and more misinformation spread.
In my opinion, this just makes that problem worse. There is often a reason that texts aren't already shorter. If the author didn't intend for you to read the details of something and instead wanted you to just read bullet points, they would have just made the bullet points themselves.
Most people have had mild to moderate illness and recovered, but the virus is more serious for those who are older or have other health problems.
The risk of virus transmission from food servers is the same risk as transmission from other infected people, but “one of the concerns in that food servers, like others facing stark choices about insurance and paychecks, may be pressured to work even if they are sick,” she said.
Tests have found high amounts of virus in the throats and noses of people a couple days before they show symptoms. Flu kills about 0.1% of those it infects, so the new virus seems about 10 times more lethal, the National Institutes of Health’s Dr. Anthony Fauci told Congress last week.
The death rate has been higher among people with other health problems -- more than 10% for those with heart disease, for example.
It actually did better than I expected with the paper "Bypassing memory safety mechanisms through speculative control flow hijacks" [0]
I copied and pasted the text from sections 3-7 (Case Studies - Conclusion) and Section 2 on its own (describes the attack)
It did pull out some important statements, better than I expected. Probably won't save me much time, but I was quite disappointed by the fact that the Advanced and Basic versions were the same for both which kinda felt a bit cheap to get the same results especially since it still cost to get that advanced result. Maybe including information about how the basic version is restricted and what the advanced does better would make it easier to know when the advanced version won't be useful.
I also tested with a random write-up I'll be covering tomorrow "Breaking the Competition" [1] I had higher hopes for this since it was more of a blog-ish post. I did get different results for basic and advanced with this one, but the result was basically non-sense, worse than expected, and worse than the paper summary.
Overall, probably not something that I'll end up using, but technical content also isn't the intended use-case which is totally fair. I'll also add that one feature that I looked for immediately was API access as I'd have wanted to integrate this into an app I use to plan episodes.
- [0] https://arxiv.org/pdf/2003.05503v1.pdf
- [1] https://medium.com/ctf-writeups/breaking-the-competition-bug...
That said. Keep at it. It seems like a viable and valuable service.
On March 9th America’s government awarded a trio of firms $39.7m to design “microreactors” that can supply a few megawatts of power to remote military bases, and be moved quickly by road, rail, sea and air. The idea of small reactors is as old as nuclear power itself.
In July 1951, five months before a reactor in Idaho became the first in the world to produce usable electricity through fission, America began building USS Nautilus, a nuclear-powered submarine.
A report by the army in 2018 said that Holos, a prototype mobile nuclear reactor, would be 62% cheaper than using liquid fuel.
NASA is developing smaller “Kilopower” reactors for space missions, designed to power small lunar outposts.
Also, when it comes to summarization- you don’t really need to infer each run, you can throw up a pretty simple caching system. Which means repeat requests are far cheaper and faster.
I used cloudflare workers as a proxy / caching layer with KV in front of an AWS lambda to do article extraction and SageMaker spinup (with a small cache on the AWS side too- to catch in progress jobs)
Are you planning to launch an API anytime soon?
* It's intended in part as a commemoration on the 50-year anniversary of the sketch, but also to draw attention to the need for a more streamlined peer review process for grants in the health sciences.
* "So, put together a Monty Python fan with a creative scientific mind and an expert in gait analysis, and this paper is what you get," Butler told Ars. Or, as they wrote in their paper, "It really is the silliness of the sketch that resonates with us, and extreme silliness seems more relevant now than ever before in this increasingly Pythonesque world."
* First aired on September 15, 1970, on BBC One, the sketch opens with Cleese's character buying a newspaper on his way to work—which takes him a bit longer than usual since his walk "has become rather sillier recently." Waiting for him in his office is a gentleman named Mr. Putey (Michael Palin) seeking a grant from the Ministry to develop his own silly walk.
* (Note: the name is spelled "Pudey" in the paper but we're going with the Wiki spelling.) Mr. Putey demonstrates his silly walk-in-progress, but the Minister isn't immediately impressed.
Advanced
* One of the best-known sketches from Monty Python's Flying Circus features John Cleese as a bowler-hatted bureaucrat with the fictional Ministry of Silly Walks.
* Waiting for him in his office is a gentleman named Mr. Putey (Michael Palin) seeking a grant from the Ministry to develop his own silly walk.
* For their own gait analysis, Butler and Dominy studied both Mr. Putey's and the Minister's gait cycles in the video of the original 1970 televised sketch, as well as the Minister's gaits from a 1980 live stage performance in Los Angeles.
* Butler and Dominy found that the Minister's silly walk is much more variable than a normal human walk—6.7 times as much—while Mr. Putey's walk-in-progress is only 3.3 times more variable.
* The sketch might be satirizing bureaucratic inefficiency, but Cleese's Minister is essentially engaging in a hyper-streamlined version of the peer review process in his meeting with Mr. Putey that (the authors concluded) resulted in a fair assessment.
I may try my hand at writing one with the reqs outlined in the top comment just for a fun coding project.
Method Not Allowed
The method is not allowed for the requested URL.