Truth goggles sniff out suspicious sentences in news | Hacker News Reader

Truth goggles sniff out suspicious sentences in news

niemanlab.org

78 pointsmjbellantoni48 comments

Too bad it employs the PolitiFact database, as that itself is suspect to having its own bias. (http://blog.lib.umn.edu/cspg/smartpolitics/2011/02/selection...)

Using a biased fact-checker to check facts doesn't really work well. Constructing a non-biased fact-checker is the real challenge here. I'm not sure it can be done, due to the great amount of spin that can be put on most anything political.

EDIT: BTW, I'm not meaning to take sides on this, just pointing out the very real difficulties in establishing a "fact-checker". It's worth knowing that others have contested PolitiFact's methodologies. If the trustworthiness of the statement is based on a person's judgement, then the fact-checker has bias.

Uh, it looks like that accusation of bias comes down to PolitiFact grading Republicans as wrong more often that Democrats... I don't want to get all political, but there's another potential explanation for that, yes?

(There's also the accusation that they target Republicans more than Democrats, but that's easily explained by their methodology, from that post: "If we look at something and we think that an elected official or talk show host is wrong, then we will fact-check it." If we posit that Republicans are wrong more often than Democrats, we would expect them to show up more often.)

Yes, but the parsimonious explanation is that neither Democratic politicians nor Republican politicians nor any other kind of politicians are particularly careful with their facts, and which you think are more prone to "lying" has more to do with the listeners than the speakers. The idea that any politician is routinely dealing in truth is laughable... and I don't mean that so much in terms of cynicism per se, but because they really have very little incentive to do the necessary huge amounts of work it would take that would bring them no gain, or even less than no gain inasmuch as it would limit their verbal weaponry.

Just as one example of the sort of thing that makes it difficult for me to take Politifact ratings seriously in 30 seconds of browsing, look at: http://www.politifact.com/truth-o-meter/statements/2011/sep/... Mitt Romney gets a "pants on fire" rating for claiming that Obama has apologized to the world for the US on various occasions. To my eyes, it's a desperate attempt to avoid the accusation by deliberately and stubbornly being as naive as possible, insisting that it wasn't anything remotely resembling an apology ("pants on fire" remember, not just "half truth", which I would actually accept just fine on this matter) because it doesn't contain the word "sorry" and never quite fully says "apology". I call it "naive" because in politics everybody knows that there's more to what's going on that just what is being said; would Politifact expect me to be naive enough that when diplomats say they had the traditional "frank exchange of views" that means the negotiations are going well, because nobody said outright they were going poorly? Of course Obama never said "I apologize for the US", but there's rather more wiggle room for interpretation than giving someone who says there were an apology a "pants on fire" rating.

And to be perfectly clear, I see equally dubious straining-at-gnats attacks from conservatives on liberals as well routinely. The only way these "truth goggles" could actually work is to draw from a wide variety of sources and hope the bias cancels.

no, the parsimonious explanation is that the person caught lying most lies most. really. it sounds trivially obvious because, well, it's trivially obvious.

sure, politics is complicated and everyone bends the rules. but that would give a "background noise" that was the same for everyone. sometimes anyone can make a mistake. sometimes anyone can get away with it. there's no bias there.

if you catch someone who lies more than the others then the simple, obvious explanation is that they lie more than the others. if you want long words, then it's similar to the difference between systematic and stochastic errors.

in this context it's a little ironic to say this, but surely only a liberal who was so completely disconnected from reality that they have no idea of right and wrong would argue otherwise: some people lie more than others; you can tell who they are because they are caught lying more. to argue that there is no right and wrong in people's statements is "political correctness" taken to crazy extremes.

of course, there are exceptions: if you only check one person then it's unfair. but as far as i can tell (see my other comment) that is not happening here.

The person caught lying most is probably the person who is the worst at allaying suspicion. PolitiFact's fatal flaw is that they only fact-check a subset of statements: those which sound suspicious.

Further, it's important to distinguish lying from being factually incorrect. A lie implies deceit, being incorrect does not.

In both this and the parent comment, is parsimonious really the adjective you were looking for? In what way is either of these a parsimonious explanation?

Simple, perhaps. Reductionist or nominal, certainly. But the cheaper, stingy explanation? I'm not so sure that was correct.

have you read that article in detail? i have spent some time on it, and am having trouble following the argument. at the start they say that approximately equal numbers of statements are taken from either side. then later they seem to be saying that a bias exists because the republicans lie more than the democrats. what i'm having a hard time understanding is why that is a bias - perhaps some republicans simply lie more?

again: it's not they are focusing more on one party than on another (they don't: "the number of public officials subjected to PolitiFact's Truth-O-Meter lens from each party is fairly even during the period under analysis"), but that they catch more lies from one party (the republicans) than the other. in particular, it appears that palin and bachmann are responsible for raising the average number of lies for the republicans.

so why is this bias? why isn't it simply that palin and bachmann are liars? what am i missing (i am not american, so don't know much of the context - i just read the article and couldn't follow the logic of the argument)?

one possible alternative is that they are ignoring lies by democrats and instead checking only true statements. but in that case you'd think there would be a smoking gun - notorious statements from democrats that weren't checked. the article doesn't give any evidence for that.

ps also, i take issue with your final disclaimer. you can make judgement calls and still be better than coin flipping. just because it's not an automated, exact science doesn't mean that it cannot edge closer to the truth. almost everything in life is not exact, but we still know some people lie more than others. one way to do this is by looking at what is given (for example, this article) and checking whether it makes sense. maybe i have made a mistake - if so, hopefully someone will correct me. eventually this thread will lend weight to one conclusion or another...

you can make judgement calls and still be better than coin flipping. just because it's not an automated, exact science doesn't mean that it cannot edge closer to the truth. almost everything in life is not exact, but we still know some people lie more than others.

I agree with this in the fuzzy world of politics you have no choice but to bring in probabilities. I would think that this system could be automated by software to aid the fact checker in coming to a decision. I would say that aggregating a politicians history of lying and using that as a weight as to whether a suspect statement could be untruthful would help.

If a candidate has a history of being less than truthful and a certainty of fact cannot be deduced one could look at weight factors such as history of truthfulness on a subject to help them come to a conclusion.

These methods and the algorithms could them be published for scrutiny. This way it is not arbitrary, it may not be 100% accurate but at least the system is open and the method reproducible. From there it can be refined to help aid in accuracy.

> so why is this bias? why isn't it simply that palin and bachmann are liars?

I hesitate to get into this, but Peter Orszag has only been tagged by Politifact once[1]. Yet at the time he was CBO director, there were many people stating that CBO was distorting long term projections to show that Obama's health care bill saved money. And recently the administration appeared to acknowledge that these projections were (at a minimum) indeed inaccurate[2,3].

  By scrapping the CLASS Act, HHS is losing about 40 percent 
  of the savings healthcare reform was supposed to generate. 
  The Congressional Budget Office’s most recent estimates 
  projected $210 billion in total deficit reduction from the 
  healthcare law, $86 billion of which would have come from 
  CLASS.

  HHS officials emphasized Friday that the healthcare reform 
  law is still projected to cut the deficit, since about 
  $125 billion of the most recent savings estimates are 
  still intact.

Now, you can determine for yourself whether you believe HHS and CBO's other projections given that this program was found to be insolvent before it was even launched. Please also note that there was no shortage of people vociferously opposing[4] the healthcare bill before passage who called out this particular provision as unsustainable due to adverse selection effects.

  The CLASS Act looks like a gift: It brings in $58 billion 
  in net tax revenues by 2019, lowering the deficit by an 
  equivalent amount because only minor costs will be booked 
  during that period. Under the CBO rules, the CLASS Act 
  technically covers one-quarter of the $250 billion 
  shortfall in funds needed to pay for health-care reform.

  The gimmick lies in looking only at the CBO's ten-year 
  budget window. The extra revenues are an illusion because 
  of the disaster lurking just beyond that horizon.

  In fact, none of the $58 billion is available to pay for 
  the House bill. The CLASS Act is so poorly designed that 
  the $58 billion reserve and all future premiums won't come 
  close to covering the generous benefits it's promising.

  Here's why the mechanics of the CLASS Act assure its 
  eventual collapse.

You may disagree, but this is one relatively cut and dried example in which the critics of the healthcare bill appear to have been proven right. Indeed, evidence-laden, numerical critiques of this particular "CLASS Act" provision were all over the web along with charges of falsehood by Orszag.

Yet at the time when it might have moved the needle, Politifact was silent on this issue. This may be what OP is referring to in terms of a selection issue. They simply did not engage the CBO's projections, even though they were far more amenable to classification as "true" or "false" than standard political statements. They were also inarguably important.

[1] http://www.politifact.com/personalities/peter-orszag/

[2] http://www.nytimes.com/2011/10/15/health/policy/15health.htm...

[3] http://thehill.com/blogs/healthwatch/health-reform-implement...

[4] http://money.cnn.com/2009/09/03/news/economy/health_care_cla...

Is there a particular statement from Orszag that you feel Politifact should have addressed?

Well, this one is interesting.

http://www.whitehouse.gov/omb/blog/10/03/04/No-Gimmick/

  Recently, a lot of attention has been paid to a claim that 
  this deficit reduction is achieved only through a 
  business-as-usual Washington budget gimmick: paying for 
  just a few years of costs with many more years of savings.

  This charge is simply false—and let’s get the facts 
  straight. 

  ...

  Third, this is not a budget gimmick.  The purpose the 
  tried-and-true gimmick described above is to make a 
  proposal that adds to long-term deficits appear fiscally 
  responsible. But if that were the course we were taking, 
  we would expect to see a large fiscal hole at the end of 
  the first decade and larger and larger deficits in the 
  second decade. Instead, over the long-term, the savings 
  under the President’s plan are expected to grow faster 
  than the costs.  So, when the Congressional Budget Office 
  is done with its scoring, we expect it will find that the 
  President’s plan reduces deficits by roughly $100 billion 
  in the first 10 years and roughly $1 trillion in the 
  decade after that.  In other words, health reform should 
  reduce the deficit by growing amounts over the long-term.

  Put simply: Health reform will reduce the deficit in this 
  decade, and it will reduce the deficit by even more 
  thereafter.  There’s no gimmick in that.

A program which was supposed to save $86 billion is declared insolvent and discontinued before it was put into production, by the people who were intimately involved in designing it. They pushed hard to pass the bill and made statements like Orszag did above.

In the interest of being nonpartisan, I think that history will look on these kinds of statements in the same way we now look on Colin Powell's presentation to the UN in the context of the runup to Iraq. At best the WMDs/savings just weren't there, and it was an honest mistake. At worst, they lied. Somewhere in the middle one might call them guilty of wishful thinking and exaggeration. But whatever you call it, it's hard to continue to maintain that those who claimed the presence of WMD/savings were empirically correct.

So you want PF to retroactively check people's predictions to see if they were correct? That would be interesting to see, but it's clearly not what they do, they address current statements based on current information.

Now was there any reason to believe that something in that statement wasn't correct when he made it? (for example was it not going to be scored by the CBO?) Otherwise I don't see the bias.

> Now was there any reason to believe that something in that statement wasn't correct when he made it?

Yes, at the time Orszag made the statement many commentators (including the CNN link a few posts up) contested the CBO's projections and said they were manipulated, with a particular focus on the CLASS Act. Orszag doubled down just like Powell did before the UN, just like the yellowcake incident.

I happen to think this specific case is fairly cut and dried, but short of Orszag admitting to a "lie" it is impossible to know what was in his mind. (Though it should also be said that Politifact does not precisely distinguish between whether a statement is empirically false and whether the speaker is a 'liar with their pants on fire').

Anyway, this is one example that stuck in my head as a numerical one which was so important beforehand, and so checkable both at the time and today, that even partisans on opposite sides might meet up to compare notes afterwards.

You are absolutely right -- and I'm not coming into this project with the illusion that any given source will be universally trusted (or trusted at all). I don't want to create a tool that people will look at and say "all righty then! Time to believe anything this tells me!" In fact, that would be counterproductive.

The real goal is a tool that will trigger critical thinking when it is most needed -- flag pieces of content that are worth spending an extra few seconds / minutes / etc looking into and then help them look into it. I want people to think carefully before infusing controversial information into their world view.

(btw I'm Dan in case you didn't pick up on that)

I've got a strong (but untrained) interest in the psychology of bias, and I don't think it is possible. Humans, even smart humans, even smart humans who study bias are notoriously unaware of their own biases and self-deceptions. We tune out information that challenges our belief system, we form alliances, we make the fundamental attribution error, and we attach meanings that tend to reinforce our beliefs.

It's really good to try to be unbiased, but understand that it's an asymptotic kinda of thing.

Very valid points. I think that "facts" in the political dialogue are an illusion and politics has always been interpreting reality. Of cause some "truth machine" could spot misconceptions of obvious facts (e.g. how many states are in the Union) but with anything slightly more sophisticated it is going to be a subjective mine field. Just take some basic question and get studies drawn up by different ideological thinktanks and I am sure they are able to produce the most beautiful statistics arguing that their ideological position always has been and always will be the right one. Especially since PolitiFact attempts to "rate" gray areas beyond just true and false their neutrality will, and has to be questioned.

There are few ideas that I see that have the potential to change the world, but I would say this one may. A lot of people are too busy or to intellectually lazy to research facts. As such they default to non-verbal ques in candidates. The strongest being confidence.

A candidate can be confident while completely fabricating information and more people will believe them than a candidate with no confidence and truth. Having a tool like this to refute the information immediately will be valuable.

The trick will be the credibility of the "truth" provided. We all have biases and keeping them out of our creations can be more tricky than it would appear. No doubt if the product gains wide traction, many will accuse it of bias the author needs to make sure that they have double and triple checked the facts so that it can withstand the onslaught of accusations that will surly come from both sides.

I've been thinking a lot lately about how technology can provide transparency to the political process. Among these ideas have been "reading aides" like this: tools that will provide verification and context for what you're reading or watching.

Another such tool I've been thinking about would show you the relationship of the writer and quoted sources (people or institutions) to the material being covered.

For example, consider the case where spokesperson for a non-profit organization is quoted defending a particular industry. This tool might tell you that this person was once an executive at a for-profit corporation in that industry or have some other non-obvious relationship.

If anyone knows of anyone working on something like this, I'd love to learn about it.

I have been working on Whybase (http://whybase.com/), which allows you to connect ideas and analyze statements made by people. You can use it to discover insights into the reasons behind why things are true and how things are interconnected.

Can you give some example ?

"Goggles" "sniff"? Do people even pay attention to their own metaphors anymore?

I've dreamed about the possibilities for automated fact-checking to improve online debate, and considered attempting to write it myself (using data sources like Wolfram Alpha and Google Public Data, and parsing natural language with a big dirty pile of regular expressions, which can't be the best approach but it's an approach I could attempt in a weekend).

For example, a program could examine a text for sentences matching "[X country] is [comparison] than [Y country]" and check that against online data sources. So if you're typing a comment into reddit or wherever, it would first check your comment for claims it can parse, and alert you if you are mistaken about something so you can change it. And if you are correct about something, but didn't supply a citation, it could add it automatically.

This could have several awesome effects:

* It would make online debates more accurate, spreading truth and knowledge.

* It might set a standard of factual accuracy; people might come to expect assertions to be backed up with citations, and regard claims as suspicious if no supporting reference is provided.

* Just knowing their facts will be checked (possibly by a future version of the software being re-run on old forum comments and highlighting inaccuracies for all to see) might make people more careful in their assertions.

If voice recognition technology improves, then this might eventually fact-check TV in real-time.

This is really awesome, but I think it would be even better if a program could be written such to flag sentences without a third party judge. Using the stylistic tendencies of lying, for example, in rhetoric. This might not be possible or plausible, but if it could be written, I think it would be at least equal to this technology.

There are at least two reasons that's impossible. First, liars are smart. Second, not everyone spouting an untruth is a liar. If a person doesn't know he's uttering a falsehood, then the fact is unlikely to influence his style — or his speech spectrum or cadence or direction of gaze, etc.

Well, I see a solution around the first reason. Just flag anything that is worded in a suspicious way. The example, "Obama has the worst ratings in modern times" sounds suspicious. You shouldn't need a fact checker to know you need to check that fact. If you could implement the same sorting method common sense has to a fact checker, you might be able to make it work.

However, I admit, you might be faced with the problem that an overwhelming number of statements would be flagged. I'm not sure how you would get rid of false positives.

Don't know why, but the page is completely broken when rendered in Opera. Perhaps the custom font?

Perhaps the page isn't broken, the browser is.

1) The user doesn't care. Therefore it is the site's problem.

2) Works fine for me with Opera (linux). so shrug.

Nah. When the browser has a single digit share, it's on the browser to make things render correctly. The user went to some trouble to get that arcane tool.

"The user went to some trouble to get that arcane tool."

Troll much?

Only when the mood strikes. It didn't above.

"Who uses Opera?" seems to me a perfectly valid sentiment in favor of not spending one's limited energies on making sure something renders correctly for it. Hell, in a cousin, the abundantly reasonable admin of the linked site reports it's less than 1% of their traffic.

Like I said, arcane. Worry about what real users use, I say.

The trolly-ness was specifically the sentence I quoted. Anyone who is already aware of Opera (more people than you might think, usage != awareness) will not have any trouble acquiring it. I have numerous non-technical acquaintances that use it on their often aging hardware. It's not like we're talking about elinks here or something.

Except it has share in tens of percents in some markets. The fact that it has 0.5% market share in US doesn't mean it has no market share at all. You are right that it doesn't make sense for US site to focus on Opera but your other statements are quite invalid.

Looks fine to me in Opera Mac too, fwiw. (I run niemanlab.org, and I confess I don't do any testing in Opera, so I honestly appreciate any bug reports.)

FYI, our browser breakdown: 31% Chrome, 28% Firefox, 15% desktop Safari, 11% IE, 10% iOS Safari, 3% Android browser, 0.74% Opera, 0.26% Opera Mini, 0.23% RockMelt (!).

Might be some userscript I have installed, thanks for the feedback.

And as a bonus, browser stats from country I'm from: http://gs.statcounter.com/#browser-SK-monthly-201010-201110-... (You can check more countries from this region, you will see similiar results for Opera (Ukraine stands out though)). The reason for that is that Opera was the best browser without any doubt (think pre-firefox2 age). Only flaw was that it was not free - it was ad-sponsored. But you could get it for free of course, and users in Easterish Europe didn't really care.

"... software that flags suspicious claims in news articles and helps readers determine their truthiness."

+1 for use of the word "truthiness". I can't believe the impact people like Stephen Colbert have on our culture and lexicon.

If you're looking for truth in media, you're looking in wrong place.

What? The article is about direct quotes from politicians. There are a lot of things you can blame the media for, but this isn't one of them.

The article is about "suspicious sentences in news", and media are news producers.

This is one thing I can blame them for, due to: media needn't be objective - that's not their mission - their mission is to make money, unless they're non-profit (how many of them is ?).

They use deception, to make you believe that news they produced is objective, and you shouldn't look for sources (which often are well hidden or absent), and assuming that news they produce is compatible with your worldview, you won't look for sources - you're their customer.

I'm sure you're able to divide media by political views they support - if they're objective why it's possible ?

The use of deception, means they're not good source of truth.

The main reason I found HN and stayed here, is its news quality (almost no political bs, and really close to the source).

So how do you propose we function as a democracy if any media source that covers political issues is so full of shit that there is no point in even considering it?

Your perspective is extreme, but it is a very important concern. If only someone could try to create additional information layers to help you navigate a medium that is potentially riddled with dubious claims. Maybe that person could call the project "Truth Goggles" or something.

The media's political analysis absolutely is worthless. If that's all that keeps democracy functioning, we might as well flip coins.

So how do you propose we function as a democracy if any media source that covers political issues is so full of shit that there is no point in even considering it?

From my perspective, for democracy, it's very important to have independent, non-profit media that provide raw stream of information - e.g. A/V stream from congress, major events etc. - that while being boring would provide unbiased stream of facts. For news - transcripts, photos, videos, documents, all without commentary. That would allow to compare "revelations" served by other media, with facts.

If only someone could try to create additional information layers to help you navigate a medium that is potentially riddled with dubious claims. Maybe that person could call the project "Truth Goggles" or something.

It would be great, if I could trust it - its sources (evidence) should be explicit.

Sounds like we do, in fact, agree. The full title of the thesis is: "Truth Goggles: Automatic Incorporation of Context and Primary Sources to Enable Critical Media Consumption"

This tool won't tell you what to think; it will merely remind you to think.

I don't understand how this can be any more effective than just cited sources at the end of a paper...

A lack of cited sources in an article doesn't necessarily imply that its false.

And cited sources does not automatically make something true.

This is like saying "I don't understand how watching a movie is any more effective than just looking at a script."

Not everybody has access to the script. Not everyone has time or capacity to take that script and turn it into a movie in their head. Scripts tell you a lot more than you need to know. Scripts aren't designed for audiences.

The list probably goes on but hopefully you get the point.

This is excellent. There has been a certain class of "debate" sites that gather opposing expert opinions on some controversy but they always had a distribution problem. The people that need this kind of information most and the least likely to go check these sites.

This is a great distribution mechanism of the output of those sites. If there is a trending, controversial statement in the news it would be great to be one click away from seeing smart people discussing both sides.

Now I want other kinds of goggles. Like goggles that spot popular Anachronisms. Or trending memes.

Reply on news.ycombinator.com