Show HN: Beautiful Visualization of Agreement Groups in the US Senate | Hacker News Reader

Show HN: Beautiful Visualization of Agreement Groups in the US Senate

friggeri.net

95 pointsfriggeri39 comments

"However [political parties] may now and then answer popular ends, they are likely in the course of time and things, to become potent engines, by which cunning, ambitious, and unprincipled men will be enabled to subvert the power of the people and to usurp for themselves the reins of government, destroying afterwards the very engines which have lifted them to unjust dominion."

- GEORGE WASHINGTON, Farewell Address, Sep. 17, 1796

Our democracy has evolved into what Washington had prophesied in his farewell address ... a system where diversity of thought and political compromise are rejected in favor of the strength of a unified political party and the power that brings with it.

When political positions consolidate into just two choices, so do the proposed solutions to the world's problems. These solutions are artificially constrained for the sake of political loyalty. Ultimately, party loyalty requires individual politicians to sacrifice their personal beliefs on individual issues ... and once those values have been compromised, the only thing worth clinging to is control.

Political parties are the weakest they've been in 200 years. The driving factor in American politics are Internet-powered movements like MoveOn.org and the Tea Party...these groups are capturing the the traditional party structure, and they are highly issue-driven. Even in fundraising, the traditional party committees are outflanked by the new Super PACs.

There is more diversity of thought and action in the U.S. today than at any other time in our history.

Op here, this is part of my PhD research. Please bare in mind that I'm French and that my knowledge of U.S politics is limited to what I can read online.

If you find any errors in my analysis or have any tips or other interesting pieces of information I'd be really happy to discuss with you.

> One unexplicable point remains in the data and that is the presence of 18 Democrats in the Republican group of the 108th Congress.

Dude, check the dates. Two words: Iraq invasion.

I thought about that, but 12 of the 18 Democrats were fiercely opposed to the invasion…

Which ones? All the ones I click seem to be in favour :)

Regardless, they might have voted against the operation but then voted for various smaller bills "supporting the troops", which is an entirely reasonable position.

I'll have to go back to my notes, but I'm pretty sure I found a lot of opponents, but maybe I'm remembering wrong. Thank for pointing this out, I'll have another closer look !

The lead up to the war was very much "you're either with us or against us". It took a lot of courage to speak against the war. Even the media had to get in line - I think all news networks hired military consultants at the time.

Over time some that voted for invasion adjusted their stance. This may be why you felt there were a lot of opponents.

Which sets of Senators were facing election that year? As a rule, only approximately a third. It would be nice to see if they were correlated to the group "switching" sides.

Which party controlled the Senate that year? It would be interesting to see if switching is correlated to party control. For instance, I see the Republicans controlled the Senate in the 108th (2003-2004).

Thanks for the idea, I didn't think of the reelection aspects, I'll investigate.

It's very good, but suffers from one basic presentational error: You've placed Republicans on the left and Democrats on the right. While I know your algorithm doesn't attempt to plot people on a particular spectrum, it so happens that stereotypically partisan conservative or liberal senators tend to be positioned at the outer margins anyway. since liberals are traditionally thought of as left-leaning and conservatives as right-leaning [1], I would suggest your diagram needs to be flipped horizontally.

1. http://en.wikipedia.org/wiki/Left%E2%80%93right_politics#His...

The visualization is totally cool, though I don't claim to understand the analysis.

...but if it helps, you made a little English error. Wouldn't even have mentioned if you hadn't said you were French :)

> [Republicans.] Most of which are either moderates or closer to the Democrats than to their own party.

> Olympia Snowe and Susan Collins are the two Republican Senators from Maine which are both regarded to be leading moderates within their party.

Should be "Most of whom..." and "...who are both...".

"Which" is only ever[1] used in relative clauses to describe inanimate objects (as you do correctly five other times in the article). Humans and animals are always either "who" (more personal) or "that" (more impersonal).

Note that's only in relative clauses-- "Which senator is yours?" is fine, but "The senator which represents me" means that I think my senator is a card table.[2]

[1] Outside of the odd dialect, of course.

[2] Not necessarily out of the question.

I'll assume you're correct in your explanation. I've never heard it explained that way.

But as a native speaker "that" sounds slightly more correct than "which" because it seems less formal, however, if I heard a non-native speaker say "which" it wouldn't sound weird nor would it be something I would have corrected them on.

Of course, with all of this stuff, people have different sensitivities-- and I know there are some dialects where "which" regularly references a person. I actually might not notice the first occurrence myself, although on reflection it looks a little odd. The second one jumped out at me as seeming pretty weird, though.

Even so, I wouldn't correct it except that, read uncharitably, referring to a person with an inanimate term could be seen as somewhat insulting. Especially if you're writing about politics, it's probably worth getting right just to avoid that impression.

In American English, which is normally used in non-restrictive clauses (offset by commas), and that is normally used in restrictive clauses (not offset by commas).

Using which in a restrictive clause like The senator which represents me... or The road which runs from X to Y... is somewhat uncommon. Both of those examples sound better in American English using that in place of which. If the subjects didn't need disambiguation, though, that would no longer be appropriate, e.g., Senator Jones, who represents me, ... or Foobar Street, which runs from X to Y, ...

As Cushman points out, though, which used on a person doesn't quite work. That can work on a person, though, but who/whom can work for people in both restrictive and non-restrictive clauses.

See http://owl.english.purdue.edu/owl/resource/645/01/ if you want to read a well-structured and thorough explanation.

You're perfectly right, I've reworked the text several times and these mistake slipped through the proofreading. Thanks for pointing them out.

Typo in the Hagel quote: it's "almost every area", not "almost every are".

Would be interesting to see PCA and CA* applied to the same data.

On a quick google trip I found

[1] Principal Component Analysis of Senate Voting Patterns (http://escholarship.org/uc/item/3xm9z62w#page-1)

[2] Shlomo S. Sawilowsky, Real data analysis; Chapter 28, Principal Component Analysis of Senate Voting Patterns (on Google Books)

* Principal Component and Correspondence Analyses respectively -- http://en.wikipedia.org/wiki/Principal_component_analysis and http://en.wikipedia.org/wiki/Correspondence_analysis

It's not the same data, but I did a PCA with the voting record of Canadian MPs. It's a bit old, so it's in flash instead of html5. You can access it here: http://www.votum.ca

I'm not sure this would give the same kind of results, it was shown [1] that Kleinberg's HITS algorithm is PCA, and HITS gives [2] hubs (nodes linked to many others) and authorities (nodes linked from many hubs). I have the intuition that this would qualify the floor leaders as authorities, but I'd have to run the numbers.

Anyway, in order to obtain groups from PCA (that is, groups which are not only "visual"), one would have to run some kind of clustering algorithm on the reduced data.

[1] http://www.uclouvain.be/cps/ucl/doc/iag/documents/WP125_Saer...

[2] http://en.wikipedia.org/wiki/HITS_algorithm

It does look nice... and it does raise interesting questions. I'm particularly intrigued as to John Kerry, John Edwards, and Joe Lieberman's presences in that anomalous 108th Congress. Can you tell us any more about how you define an "agreement group"?

Visually, the result reminds me of [http://xkcd.com/657/] - is this sort of diagram a recognised concept in data presentation, or a new idea?

[edit - hmm... http://en.wikipedia.org/wiki/108th_Congress]

Those three all voted to invade Iraq. There were probably a number of related votes for logistical and budgetary arrangements to start that operation.

Yeah, hence my edited-in "hmm". When I first wrote that post, I hadn't yet hit that Wikipedia article, and, as a non-USian, I don't know what years which Senate sat for off the top of my head.

Makes more sense now.

I was not aware of this xkcd comic, funny how things seem to pop out. My inspiration was more in the line of Sankey diagrams. Among others, the famous map of Napoleon's invasion of Russia by Minard (http://en.wikipedia.org/wiki/File:Minard.png). I've called this representation a "groupflow" but any name would do.

The notion of agreement departs a little bit from the classical fraction of common votes/total votes and instead is based on a vector model of votes, using cosine similarity as a value of agreement. The agreement groups are computed by heuristically optimizing a quantity called the cohesion and which turns out is really correlated to the way people perceive social communities (the old preprint of the first paper: http://arxiv.org/pdf/1107.3231.pdf).

Minard's graph is brilliant.

On my to do list: a graph that shows the loss of voters casting ballots. I needed a way to show how postal balloting disenfranchises voters (what it gives, it equally takes away).

Voting age population -> registered to vote -> voter received ballot -> returned ballot -> elections rcvd ballot -> signature verification -> challenged ballots -> overvotes/undervotes, voter intent, adjudication -> tabulated.

Actually, now you mention it, I think I remember seeing the author of XKCD mention that he was inspired by Minard's diagram.

I'm afraid your paper's too much for my brain on a Sunday morning, but the diagrams and typesetting look really well done :)

Looks really great and seems like a useful tool. My only real complaint, is that I use a high sensitivity on my mouse and I found it difficult to highlight a line. If you could make the lines slightly thicker, it would be easier to select them.

For what it's worth, I thought the gray background bars were far too light - it took me a while to find them. I also couldn't intuit what the branching lines were all about.

Also, Chrome had some difficulty with typesetting, though it's possible that's a localized issue.

*edit: I figured out why the boxes seem so light - it's because I view my screen at a slight angle. From a straight angle they show up fine.

Sorry about that, I should have clarified that each Senator could be in different groups, hence the branching. I'll add a little darkness to the background bars.

What kind of typesetting issues did you have, and with which version of Chrome?

Here's a screenshot from my Chrome 18 on Linux: http://i.imgur.com/vPwl8.png

The visualization is quite nice, and it would be useful to link to or include some kind of short description about the algorithm and visualization - to give a different example, it looks like the groups are scored numerically but the senators are simply ranked, so I wonder if the algorithm found more than two groups per congress.

I have absolutely no idea why it renders this way on your Chrome, maybe an issue with TypeKit?

About the visualization, it was done semi-manually with a tool I've developed to heuristically minimize the crossings, but their is no way to obtain this kind of layout purely algorithmically (I'm pretty sure it's NP-hard).

  it looks like the groups are scored numerically but the senators are simply ranked

Not quite sure what you mean. The groups were positioned in order to have the one with most Republicans on one side and the one with most Democrats on the other one. There were consistently two groups for those sessions, this I can assure you. The order of the senators inside the groups is arbitrary, in order to attempt to minimize the number of crossings and provide a readable output.

Ah, I assumed each senator was placed according to some measure of extremism, though in hindsight that alone should have produced crossings.

To support this interpretation, I noticed that the groups are not aligned - see for example George Allen's swerve in the 108th Congress. I suppose I was expecting it too, from having seen some visualization that tried and failed to find a tea party cluster.

Simon Jackman calculates ideal point estimates based on non-unanimous roll calls: http://jackman.stanford.edu/ideal/currentSenate/x1.pdf

I think his R source is available too.

Good stuff. Can you talk more about the algorithm?

Can you use this to detect issue clusters rather than voting clusters?

I have tried to build subgroup-in-graph detectors with only partial success.

Thanks, the algorithm is a heuristic which optimize a quantity I've called social cohesion (http://arxiv.org/pdf/1107.3231.pdf), the article presenting the algorithm itself is currently under review, but basically: first it computes the best possible group for selected edges in the graph, then it merge groups which overlap too much. The whole difficulty is to identify how to select those edges, how to expand around them and then what does "too much overlap" means. I'll link to the preprint as soon as I upload it to arxiv.

What do you mean exactly by "detect" issues? C^3 should work if there is a notion of transitivity in the network (that is, if A and B are similar and B and C are similar, then with very high probability A and C are similar).

Great stuff. Thanks for sharing.

It'd be great to apply your algorithm per issue.

During Camp Wellstone training (grassroots political organizing), they taught us a tool called a Power Map. The two axis are yes v no and influence. You graph each person voting. Then you draw arrows to show who influences who. It helps to identify where you should focus your lobbying.

One point that stood out is that a Power Map is needed to be done for each issue, because many many votes don't come down on partisan lines (eg local and state, ymmv).

You have a bipartite graph of bills and representatives. You use co-voting for the similarity metric to cluster representatives. Flip that around and cluster the bills using similarity of voters.

Best use of SVG I have seen yet!

1) Many senators shift to the left side (more conservative) as time goes by

2) It's especially true about republicans

Reply on news.ycombinator.com