Show HN: Python package to collect normalized news from almost any website
github.com
github.com
By this you mean ... Fetch RSS feeds from the websites that we've hardcoded into a Python library?
It seems like the work here was in collecting and cataloging a lot of RSS feeds. You don't seem to be able to arbitrarily "collect news from any website".
My approach has been, if we can't get neutral/objective coverage, getting comprehensive, 360 desgree coverage is a good alternative.
These days, news bias is not only in the way a story is covered, but also which stories are covered. So, maagnit automatically collects stories from the left and right.
Is there anything you’ve learned about “the left media” and “the right media” from doing this? Do you think your sources are equidistant from “the centre”?
Not to mention americans have demonized "the center" as some "if you're somehow trying to consider all the facts you're a coward who can't decide" type of thing. The two sides being "at war" drives TV/website engagement and that's all that matters to the people writing the headlines.
It just so happens that currently, one of the two "sides" relies heavily on disinformation; so anything that tries to fight disinformation (including remaining impartial) is that side's enemy. So those things end up being considered "left-wing".
Truly, the united states has four political parties: The Media Left, the Media Right, the Political Left, and the Political Right. Nearly every american you know is part of the first two; the last two don't make for good TV.
We already see this in the media. You have the host and two people. Lets hear what person A says, lets hear what person B says. Who's right? Impossible to tell, otherwise you must be biased. And no mention that person A drew conclusions on claims which were mostly verifiable facts, whereas person B drew conclusions on entirely made up claims.
In fact, this is a known fallacy and has a name: argument to moderation.
https://en.wikipedia.org/wiki/Argument_to_moderation
Some times person A is 100% right and person B is 0% right. The thing is that to be able to tell the different takes effort in educating yourself. And the minute you try to outsource this to someone else, they will use that trust against you. But calling it 50-50 is not a solution, it's not even an approximate solution.
> The thing is that to be able to tell the different takes effort in educating yourself.
Actively researching both sources and stories, and not being a passive consumer who expects someone else to spoon-feed you a solution.
Nah. My view is that news exist for entertainment. That might not be their stated goal, but it's why people consume them.
The Guardian (yes, ironically a provider of "News"), published an extract from the writings of "The Art of Thinking Clearly" author with some decent enough bullet points on this that are difficult to argue with:
https://www.theguardian.com/media/2013/apr/12/news-is-bad-ro...
I think there's a threshold of distance and size where it stops being useful to the average person. Too far away? Can't do anything about it? Better not weigh down your mind by getting stressed about it.
[1] http://press-pubs.uchicago.edu/founders/documents/amendI_spe...
If you read garbage publications, of course you will get garbage arguments and opinions.
US news channels are a horrible example, they pick someone to represent the most extreme views and have them fight on TV to entertain viewers.
But reading two articles on the same topic by two sources on different ends of the political spectrum and with decent quality and can be very enlightening. (emphasis on quality)
Journalists and politicians alike getting drawing incorrect insights from data.
Meanwhile so little of the data surrounding coronavirus is comparable. You have countries only recording cases if the patient was hospitalised (not if they tested positive) and cause of death reporting is a minefield.
So "verifiable fact" is just as easily weaponised / politicised as bullshit. At least bullshit is easier to debunk...
We had this almighty Imperial Model projecting 500k dead meanwhile Sweden just did their own thing.
I don't really have a leaning to the policy setting but it feels that once you wrap some information up with numbers, maths and scientific authority you can parade subjectivity as fact anyway too.
I guess I'm a little jaded after seeing politics and business use spurious pedestalise data which really amounts to numerology when you start peeling back the layers and thinking critically.
That someone made assumptions and drew conclusions might be a fact. But the assumptions might or might not be facts, and the conclusions certainly aren't facts. As a general rule, if it's in the future, it's not a fact.
In the very strictest sense of verifiable fact - boolean true/false statements with irrefutable proof, then yes I think fact trumps all.
Unfortunately I think that in reality, very little of what we think of as fact falls under that very narrow definition. And my example was merely trying to illustrate that the entire field of medical statistics and epidemiology is filled with pitfalls, given that many of the numbers that are reported themselves have a tonne of nuance and context baked into them.
And I suppose any discussion which focuses solely on the numbers associated with the current pandemic are going to be extremely lossy as the numbers are a very low fidelity expression of the "truth" (in the idealistic sense).
What does this have to do with facts? I guess I feel that culturally we attribute the trust to things we consider "facts" at quite a superficial level without digging deeper.
I'm not sure if you are just throwing this out there or if your intent is to hold up Sweden as an example that contradicted expected results with respect to models/social distancing requirements/etc.
If the later, then think Sweden is not a good example of such a contradiction: With the comparably minor-to-moderate social distancing, they have roughly about 2.5x the cases per capita as their two neighbors Finland and Norway, and roughly 6x to 7x the deaths per capita.
As you said, the data from country to country is hard to compare, but at the very least Sweden should not be the poster-child for low social-distancing mandates.
Best guess given herd immunity kicks in at about 85% of pop infected is about 260,000 deaths with a 1% mortality rate. With these figures 500,000 deaths is a 1.9% mortality rate - high but still possible.
contextualise everything with this 5% figure - the US, and everywhere else, is much closer to the beginning of this than to the end.
And it seems like your argument falls under plenty of its own fallacies.
Perfect solution fallacy https://en.wikipedia.org/wiki/Nirvana_fallacy
And some might argue straw-man fallacy given that op never suggested that truth would lie 5050 between the two.
Debate allows us to see two opposing views and draw our own conclusions; not perfect but a whole lot better than nothing.
Fixing first past the post voting is not some magic cure-all.
Two party systems also are built on coalitions. It isn't as if "Democrat" or "Labor" is a unified ideology.
I'm all for MMP but I don't think it will fix any problems in American politics.
Just introducing MMP would add another veto point in an American political system that already struggles to actually accomplish anything.
I feel as a world citizen but I am German; I know the system that was built after the bitter lessons of our history. A proportional representation (with an x% threshold to limit the number of governing parties) is really key to preventing two large parties from dominating everything and becoming the only rivals with the resulting polarisation of the political discourse.
(Free higher education would then be the next step.)
Nothing is never neutral so it’s best to show different sides of issues to make readers figure it out themselves (as easily as possible, but not as opinionated as popular news outlets)
There aren't usually 2 sides to a debate. The number of sides varies wildly and some times is even 1.
All sides on a debate are not equal. There are perfectly fine reasons to honestly disagree about issues, but there are plenty of dishonest ones, and there is way too much dishonest coverage of things on the news; there are plenty of uninformed people with opinions that don't hold any water; and there are plenty of people just not interested on truth.
Always putting 2 sides on equal footing in a debate is dishonest. It's made to have the appearance of a honest debate, while suffering from the same problems of a single sided view.
I find most media biased towards conflict in that regard. The primary goal is to trigger emotions like rage and fear.
So for my self I find I’m happier not consuming “news” at all.
This was extremely harmful to the overall mission of informing people and created a false balance between sides which are not equally valid. Today, this is almost universally seen as a failure of journalism.
Given the lessons learned, I wouldn’t be so quick to replicate that experience with a website.
The internet and social media allow you to ignore the correct conclusion because it makes a person uncomfortable with the idea that they could be wrong when everything they read reinforces that incorrect conclusion.
By far the worst actor I've seen is (old media) Rush Limbaugh. It's hard to believe that Rush could believe that America faces any threats (e.g. war, crime, economic, environmental) other than the "Democrat party." You'd imagine that he wants to see a system more like Japan (liberalism with one party) or China (one party, no rights, no dissent.)
More generally I see the problem of "spamming the agenda", combined with the "if it bleeds it leads" tendency of the media.
Before Trump ran for president CNN's ratings were in the toilet, and it lumbered on between school shootings and MH370. To report on things like that you might have to take the Redeye from Atlanta to a flyover state, but today you can sit your ass in Atlanta or Washington DC and just cover 'what trump said', and 'what somebody said about what trump said'.
Since the election, the news has seemed to be "broken" in the way that a "clock" breaks. Arstechnica's Dealmaster always has a sale on Amazon Fire devices, Anker power banks, Xbox Gold, and whatever craptop Lenovo has a surplus over. The banner on CNN often seems to go unchanged for weeks. (e.g. "TRUMP WANTED TO FIRE MULLER")
COVID-19 was the first real news in a long term, but there is an obsession about "what trump said" compared to the other 300 million Americans. Trump doesn't really care if it is good or bad news, he just wants to be in the news, and if he held a press conference and nobody came, that would bring him to tears.
Similarly there is always some article pushed by a right wing group that says there is too much occupational licensing in the U.S (maybe true, but the same editorial turns up every month as if it was fresh) or that there is a trade-off between economic efficiency and inequality. (e.g. the evidence is that if you have too much inequality people tilt the scales, rich people buy T-bills depressing interest rates, and try to keep the action for themselves and their children... One of those reasons why Stalin invented the Purge)
Showing which news sources employ the most underhanded rhetorical devices may be a positive goal in itself, but it doesn't, by itself, help the audience make their own determination on an issue. Even more of an issue is that a viewer's determination of which source is more willing to mislead or omit relevant details is much more likely to be influenced by prior opinion than by the content of either source.
Basically, the problem isn't, in itself, biased news sourced, its that the format is fundamentally ill-suited towards giving individuals enough information to come to a reasonably well-supported position on just about any topic of moderate complexity. Further take any topic that appears to be of simple complexity and scratch the surface a bit and there's a decent chance it will turn out to be not so simple.
Even if putting that aside, the utility of bringing to light underhand tactics isn't meant to be used in and by itself but instead serves as one of many aspects of debate to help decide what is right/wrong true/false.
Regarding the poor suited nature of news for getting the full story across to reader I fully agree but again just because a tool isn't perfect, it doesn't mean it gets cast aside; more perspectives (and these are mainstream organisations) on a subject doesn't hurt at all.
I don't have an answer on the issue of news organizations being poorly suited here. On the one hand, there is an appeal to your the idea you convey that something is better than nothing. However, that status quo is also what has lead us to the current situation. There is a correlation with the rise of 24-hour news networks and the internet with the increased vitriolic, polarizing, and propagandist tone things. The need to fill air time was a big part of that. I don't wholly think that was the cause. There was some trend in that direction already:
Note to readers: This next part is not intended to cast blame only in one direction. It is simply one concrete example of the type of things that became commonplace.
Around 1990 Newt Gingrich penned a memo for titled "Language: A Key Mechanism for Control" It went on to explain how language could be used to manipulate people, complete with a guide for how to use demonizing dehumanizing language against political opponents. Over the years it was systematically disseminated through his party, and when Newt became house Speaker around 1995 he literally made it required reading. Shortly after is around the time that the term "liberal" went from being a fairly neutral description like "conservative" to being a hated moniker for a political opponent. (Though right-wing, alt-right, etc., fill that purpose. now for the other side)
A journalist’s job is to try to present the truth, not create false equivalence.
The level of knowledge needed to adequately parse an expert's opinion, evaluate it, check its assumptions, research the evidence supporting those assumptions, follow up with research on the validity of that research... well, it takes a lot longer than a TV segment. In a TV segment like this, the "expert" with the better grasp on rhetoric and rhetorical devices "wins" in terms of audience agreement, and factual & verifiable basis of opinions is a distant second in terms of influence on audience agreement.
Or strike that: it's probably a distant 3rd: 1st place influence is whether or not it agrees with a viewer's current opinion. 2nd would be rhetorical ability, 3rd would be any actual evidence.
So, by all means let the facts speak for themselves-- just don't let yourself believe that is readily possible in a TV segment like those referenced.
Is there data that show CNN debates on climate change correlate with a decrease in belief in climate change? This sounds like folk lore rather than reality to me, but I could be wrong.
> Today, this is almost universally seen as a failure of journalism.
That is a shame.
Compare this to MSNBC and Fox News whose business models are really the same: cater to their viewers’ dogma/bubble and reinforce biases, and don’t try to introduce new ideas because it is bad for their business.
Usually print media is considered higher quality journalism than TV media. There are numerous studies that show reading print media leads to people being more informed than just watching TV news. Whether this is correlation or causation I can’t really say.
My understanding of both Fox and MSNBC is they are entertainment programs, not so much news programs. I’m not sure it’s fair to compare those to anything in print, nor do I think it’s entirely possible.
A good news article might present both sides of a story, but try to fact-check claims by both sides, and bring in supporting evidence. That’s the key differentiating factor of quality journalism - helping bring facts to the forefront rather than let people on either side spin the facts for their purposes.
I'd like to see a news app that has the facts and sensationalism separated, and then a reference to a more relevant fact/story that relates more directly to what a particular spin was getting at. So I guess feed their bubble with more accurate stories.
TV on the other hand is like SaaS - continious integration and deployment. Release first and apologize later if needed.
Doesn't CI rely on robust testing to ensure buggy software doesn't make it into production? I'm not sure the fire-and-forget of the 24/7 news cycle lends itself to a comparison to continuous deployment, as there is a strong bias to 'deliver now' given competition to break the news first, and it's hard to build in 'automated testing' (vetting?) of content.
No, maybe to you, but to vast amounts of people, millions of people, they watch a singular source for news and to them, that is the news.
In the US, since you’re referring to them, the news legally used to have to allow all sides of an issues to present their views. In the 1980s, the laws changed, and that is how news became so one-side.
My point is that the majority of people actually watching these news outlets see it as news, not entertainment or propaganda.
[1] https://en.wikipedia.org/wiki/Voice_of_America#Smith–Mundt_A...
—- EDIT: Reviewed your CML link from spring of 1990, interesting read, but really fails to do critical analysis of the specific regulatory changes by name, what impact changes had, etc. Beyond that, stating obvious, it fails to cover all the material events since 1990.
The question is 'How far can you go?'
For example does it make sense to present advocates for science and pseudo-science on the same level, with equal weight?
Putting a Biologist who explains Evolution on the same pedestal on equal time as a Priest who believes in Intelligent Design creates a false sense of balance between these viewpoints, even though these viewpoints are fundamentally different in their very basic quality.
You can spin this further to illustrate this point: racism, religion, false promises, lies, mental illnesses.
And even more importantly do you realize that in such world it's far more likely that only a priest will get invited to a TV show, not the biologist? Because that's the "how far" that I'm afraid of, fundamentalism that believes that the only way is to force "the truth" upon people, and it's of course always their truth.
For example: Given two commentators, if one is arguing that TV boards and government committees are supposed to be populated by and representative of the general population and the other arguing that TV boards and government committees are populated by crab people secretly supporting the illuminati; the discussion should be about how effectively TV boards and government committees are being filled by the general population and how effective they are at articulating societal consensus related to important issues.
In the "gotta hear both sides" world, rational discussion is dragged off the table by the notion that every issue has two equally valid sides no matter how absurd one "side" might be.
But were revealing a more fundamental issue here: Where are the checks and balances of the media?
As it stands now, we've got two possible solutions:
1. Censorship, which I find dangerous and shortsighted. 2. The media itself, which results in a 'who yells the loudest' kind of culture.
This issue bleeds into all kind of problems with propaganda, advertising, fake-news, bias, ideology, lies, scams, click-bait and other kind of bullshit.
In this discussion we've been talking as if there was some kind of sane regulation of these things. But there isn't really.
All of this stuff erodes trust and creates trenches. Sometimes it feels like it is getting worse. People have been talking about how Google Search is getting weaker on this site. There is so much more noise and bullshit today than there ever was, because we're accreting information w/o distinction.
There are projects, which try to make fact-checking easier for example, new kind of platforms and ideas to foster real debate. But those things are still on the fringe.
This is a massive, important and unsolved problem I think.
Have them both on, and then probe, and even make a decision. The viewers can still choose, but it is much more of an investigation than politely taking turns.
But ruling out absurd ideas is incredibly important.
If you want to present a discussion on "Selecting sides for the news: How much restriction is too much?" Should the sides be "Industry consensus vs societal consensus" or should you make sure that the "Any censorship is intellectual theft and every news outlet must comply with individual requests for people to state their positions on the topic" side is treated the same as the first two?
There are some good faith and sound mind tests that should be applied here but if someone wants to genuinely make a case for choosing a political leader via athletic competition (which has no basis in history at all, nope) then why not hear them out? What are you afraid of?
people can have different opinions on how much of it is caused by people.
for example moving to biofuels is being considered a big fuckup and this was done by the biggest supporters of imminent climate catastrophe.
There are reasons it is no more global warming but climate change.
more progressive people tend to move faster to progress but at the same time they make more errors while doing it..
The modern spin on The Advocates is Intelligence Squared[2].
[0] http://openvault.wgbh.org/collections/advocates/full-program...
The longer format and strong moderation really lend themselves to interesting discussions, and you can learn a lot. One of my favourite debates of this format (though not IQ) is Peter Thiel vs Eric Schmidt (then executive chairman of Google) arguing about whether Google was still capable of innovation [2012]:
Basically: airing of unscrutinised claims is propaganda, not news, even if done with both sides.
Broken because the talent of listeners to fully evaluate every statement is not distributed evenly.
Germany and Europe learned this the very hard way after WWI and they drew consequences after WWII.
Yes, I fully agree with you: not all opinions are created equal. Because some are scientifically accepted while others are scientifically rejected. Some are philanthropic, some are antisocial.
And we have to take a stance if we want a more humane World, and media have to preselect if we don't want war.
Anytime people argue about having a authoritarian dedicator style of hard line "DO NOT QUESTION THIS EVER" should remember you might end up on the other side of the fence.
We should always listen to the other side. History has shown people can believe ridiculous things for a long time and treat it as common sense.
Otherwise you end up with the “ministry of truth” — and we all know how that worked out.
A memorable example, in Ireland, during the campaign for the referendum to legalise same-sex marriage, was a radio news article where they were talking to someone in the hotel industry about the potential impact on the industry from wedding bookings. Because of the balance rule, they then had to have on one of the three available complete lunatics they had for all discussions of same-sex marriage to rant about how the gays were going to transgender all the unborn babies or something (due to all political parties supporting the amendment, and the Catholic Church staying out of it, there were a very small number of public figures available to go on the radio/tv for 'balance', and most of them were conspiracy theorists).
Some broadcasters have recently changed the rules around certain subjects; for instance the BBC no longer invites climate change denialists or creationists for 'balance' in most contexts.
I'm uncertain how I feel about it. It can produce truly bizarre results, like the marriage one I mentioned, but it does seem useful in some cases. A major concern I'd have about it is that it certainly _is_ used to spread misinformation and conspiracy theories, especially where one side of the argument is "basically everyone" and the other side is a weird fringe, or where the natural proponents of one side are sitting it out (common on social issues in Ireland, where the Catholic Church is skittish about opposing liberalisation too visibly).
Suppose you have a bunch of sincere answers of 1867, 1957, 2101, 2057 sweets, and then someone comes along with an answer of two million. They get a bunch of astroturf accounts, a few celebrity retweets, and suddenly thousands of people are claiming the answer to be two million, skewing the average up. They can even start using the fact that the other answers are different from each other and the average as evidence of their bias!
It is extremely difficult to determine honest effort vs dishonest spin, but it's absolutely vital to the process because you have to exclude the spin guys and all their sockpuppets before you do any kind of averaging or consensus.
No human is truly objective, so being transparent and honest about biases, and making good-faith efforts to be fair-minded, is likely as good as it gets.
I was excited when @oceanbreeze83 said:
> For any news junkies heres, I've built https://maagnit.com which gathers both Left and Right leaning sources [for any story] and displays them altogether on one page.
But unfortunately, I see nothing that does that on a per-topic basis.
> Left, Right & Center podcast also tries to address this. They pick a topic and try to get both sides of the argument. https://www.kcrw.com/news/shows/left-right-center
It looks like these guys are actually taking a shot at it, but then if I pick a topic that I know is subject to extremely propagandized (framed) reporting, ObamaGate, something that I happen to know a fair amount of lower-level detail about (but for from everything, it is an insanely complicated topic), and read their overview:
>> https://www.kcrw.com/news/shows/left-right-center/obamagate-...
>> President Trump is very upset about Obamagate. It seems to have to do with his former national security adviser, Michael Flynn — who the president fired after he lied to Vice President Pence and the FBI, and who pleaded guilty to charges that the Department of Justice is now seeking to drop. Is this a really important political issue? Or is this just President Trump’s effort to talk about anything besides the pandemic.
...I don't get a very strong feeling that what follows is going to be a sincere effort at truly "getting to the bottom of it", as much as is possible.
But this is just an intuition, I'd have to listen to the whole 55 minute talk before forming a tentative conclusion.
I find /r/NeutralPolitics to often be quite good on many topics, due to their very well thought out approach:
>> What is Neutral Politics?
>> Neutral Politics is a community dedicated to evenhanded, empirical discussion of political issues. It is a space to discuss policy and the tone of political debate.
>> Is this a subreddit for people who are politically neutral?
>> No - in fact we welcome and encourage any viewpoint to engage in discussion. The idea behind r/NeutralPolitics is to set up a neutral space where those of differing opinions can come together and rationally lay out their respective arguments. We are neutral in that no political opinion is favored here - only facts and logic. Your post or comment will be judged not by its perspective, but by its style, rationale, and informational content.
>> Neutral Politics is strictly moderated. Our full guidelines are here: https://www.reddit.com/r/NeutralPolitics/wiki/guidelines <--- Very much worth a read for those who are genuinely interested in learning more about how a "purely rational" society/organization should approach controversial topics. I would say that this is "how Journalism should be done".
Unfortunately, they seem to have only one post on the ObamaGate topic.
https://www.reddit.com/r/NeutralPolitics/comments/gngtrw/wha...
The submitter's question (see link for specifics) seems like an excellent way of approaching the question, in that he provides an example of how each side is framing/spinning the story, and then proceeds to ask a fairly awesome 5 part question on the matter. The post doesn't have much in the way of comments unfortunately - it's only 21 hours old, so maybe some will roll in eventually, but I suspect we'd have seen something by now.
Regardless, the fact that someone recognizes the true problem and is trying to do something about it, and there are 295,123 subscribers to the subreddit, is quite encouraging. But I wouldn't get too excited about the notion that this grassroots effort subreddit will get enough traction and subsequent publicity to make any serious change in the world. For that, I think any initiative needs help from famous people who will repeatedly promote it via their social media channels. But my intuition tells me that most people are so unconsciously biased, that they would be reluctant to do this, as (I speculate) the mind will sense significant risk in promoting an unbiased platform, and therefore decide against it.
I think this is the very same underlying subconscious phenomenon that @TulliusCicero and @dang are talking about in this post, but from a bit of a different perspective (italicized emphasis mine):
https://news.ycombinator.com/item?id=23227017
>>> (@dang) The underlying phenomenon, I think, is that people feel insecure in an internet forum, especially a large one, because the sheer quantity that shows up there is bewildering and our wiring did not evolve to process anything like that. Instead of seeing it as a statistical cloud produced by thousands of people (which is what it really is), we interpret it as the productions of a small group of individuals (which is how we're wired to see the world). Since that's such a distorted interpretation, those imaginary individuals seem weird and dramatic in our imagination. Either they seem super smart (because of all the information we had no idea of), which makes us feel dumb, or they seem monstrous (because of all the views that seem outrageously wrong or offensive), which makes us feel surrounded by enemies, if not demons—and so on. Because these feelings are uncomfortable, we end up creating an image of the community that we can diss in order to restore our sense of equilibrium towards it. The problem is that if everybody's doing this (and I think we all do it to some extent), it makes community really hard.
>>> (@dang) There's a long sequence of past comments on a related mechanism here: (see link above for two searches he references).
And also, while /r/NeutralPolitics have wrung about as much value as possible out of the generic Reddit reddit technical platform, I suspect doing this "right" is going to need a completely new platform, designed from the ground up specifically for this purpose. I assert that this is the most complicated problem mankind has ever had to address, it makes flight, putting a man on the moon, or splitting the atom look like a walk in the park. But if we ever hope to solve it, people will first have to realize the magnitude and complexity of the problem they are dealing with.
If we ever want to get the current state of affairs on this planet sorted out, I think some truly(!) independent organization, and platform, that is completely controlled by "the people" is an absolute pre-requisite. But I am not terribly optimistic that we will ever get one, at all, or that gets enough traction to make a difference. Just taking many of the comments in this thread as an example, it seems clear to me that there is significant intuitive opposition to a purely fact-based, freedom of speech approach - by this I am referring to how so many people frame/conceptualize this approach as a ~"false equivalence", that it gives the two sides equal "respect", something that does have some truth to it, but the manner in which people describe that problem is typically by picking the most extreme strawman example they can conjure up (exposing their subconscious bias in the process, imho).
Just a few examples (chosen at random, not the worst of the worst by any means):
> When I was a kid CNN used to do this: they would invite a single climate scientist and a single climate change skeptic and have them make their points on equal time. CNN would then say “who’s right? You decide.” And the program would end. This was extremely harmful to the overall mission of informing people and created a false balance between sides which are not equally valid. Today, this is almost universally seen as a failure of journalism.
> "Of course there are reasons why a gubernatorial election should not be decided by a ski race, but are there also reasons why it should? For the sake of fairness, we’ve brought in two experts with opposite opinions, who will now have equal time to just say those opinions, because that’s what news is."
> For example does it make sense to present advocates for science and pseudo-science on the same level, with equal weight? Putting a Biologist who explains Evolution on the same pedestal on equal time as a Priest who believes in Intelligent Design creates a false sense of balance between these viewpoints, even though these viewpoints are fundamentally different in their very basic quality.
IMO, Media Bias Fact Check [0] is on of the better external rating sources if you haven't come across them before.
My strategy for picking headlines is to let it happen democratically on Reddit where the bias is explicit in the community rules of the subreddit. I have not yet found a great "center" news source or community.
I really like your news layout and the scrolling headlines at the bottom!
The idea that our options for Covid are: 1) open up the economy and let people die until we get herd immunity 2) Keep every one home until we starve or get a vaccine.
The correct answer may be complex like Taiwan's 124 point plan https://en.wikipedia.org/wiki/COVID-19_pandemic_in_Taiwan
The real purpose of this post is to get traffic for their paid news api.
Also this closed issue is hilarious! https://github.com/kotartemiy/newscatcher/issues/3
It's quite easy to get "good" extraction for large numbers of outlets/articles without a massive amount of special-casing, as news articles are nearly universally marked up with RDF metadata (partly for Google News's benefit). Article discovery, and perfect parsing, is quite a bit harder. I ended up rolling a new Scrapy project with site-specific parsing code for an academic project as I had quite specific requirements.
I haven't pulled any articles in a while, so it's a little outdated but I love newspaper3k.
Though thousands of ppl found it useful
If that's not the case, the larger implication here that the news is free if it's in a format that is not as widely used (RSS) compared to what the mass populous uses (mobile browsers/app).
Cool library, thanks for sharing!
Also, the Terms of Service, Privacy Policy, and GDPR Policy links in the footer of your site don't work. They all have empty hrefs.
For use outside of python, here's a gist with a sorted list of sites and a sqlite dump of the site data: https://gist.github.com/tyingq/8e921eed10bf2ecf9c40ebdd70ff1...
Also, I was looking at https://newsapi.org. How does your python API compare? I see that in newsapi, it also get old articles; do you implement similar features?
Newscatcher API is a product that allows you not only get the latest articles but also search by keywords, topic, country etc. Basically, the main feature is that you search for articles that contain a specific word or phrase. This can have an added value for professionals and companies. There will be a free plan for developers with limits and chargeable for more heavy usage.
Compare to newsapi, we are less expensive for the content. We will not be able to get old articles from today. We began to stock data couple at the release.
Hope I answered all of your questions.
Disclaimer: I am a co-founder
[1] https://currentsapi.services/en/product/price
Free to try here: http://ftr.fivefilters.org/ and code for a slightly older version available here: https://bitbucket.org/fivefilters/full-text-rss/src/master/
Lately I've starting using EpubPress[1] to grab full-text articles and generate an ePub, which happens every night via cron. Then I can get a full digest on my iPad over sftp at my leisure. Sadly EpubPress is not very sophisticated, sites like Bloomberg or ArsTechnica return "are you a robot" challenges which it can't bypass.
I wish there was some kind of community driven library for retrieving full-text articles from common sites. In my vision of how that would work, users would contribute hand-crafted Selenium scripts to download and extract the article text, bypassing the bot-detection for each site. Then something like EpubPress would work a lot better.
The "modern web" just has too much junk to be interesting any more. Sometimes news sites publish articles I would like to read, but I'm not interested in dealing with 1000 different implementations of crappy mobile UIs, advertisements, animations, etc. I know reader view exists, but you still have to wait for the page to load, and it doesn't work very well for some sites. For the sites it doesn't break with, the experience with EpubPress is much better.
https://www.cnbc.com/2020/05/17/broken-internet-ad-system-ma...
At the moment the only real solution for this seems to be using your own RSS-Reader (Tiny Tiny RSS for me at the moment), which has the disadvantage of being limited to RSS-sources, as also not allowing much filtering and processing. But with more and more sources moving avway from RSS, I want something which can be fetch alternative sources and integrate them into a unified interface.
In best case it would be even work with any language, be it a shellscript or python, ruby or even java.
We collect tons of news data and let you query it with an API
How does it work internally? Is it downloading the news from a RSS or is it crawling the content of the website? Or is the content coming from an external service?
How are the feeds selected? Can we add more? who is maintaining them (in case the data is crawled)?
Thanks!
The package itself is nothing more than a SQLite database with RSS feed endpoints for each website and some basic wrapper of feedparser.
This package seems to do the first part, collecting and parsing, but lacks the interface. So what is the point of it?
And if this is maintaining it's own config for sources, can I even add my own sources? Or is this just an elaborated OPML-file with attatched business-logic?
I read your blog post about how open-sourcing helped you find testers, which is a great step for such tools.
* Shameless plug *: Our web service, Feedity - https://feedity.com, helps create a custom feed for any news webpage, via a point-and-select feed builder and REST API.
I'm still learning how to play with Python and Redis, and it's not perfect, but would love feedback.
Not to be unkind, but you should fix that. Instead of being incentivized to pay you, my only instinct was to close that page asap.
By newscatcherapi.com (this package is fully self-sufficient, you can just use it. No dependency on external services/API)
Created by <api provider> but you do not need anything from us or from anyone else to get the software going, it just works out of the box.
Is there a list of supported websites?