Syntactic: A lexical categorizer with a pretty visualization | Hacker News Reader

Syntactic: A lexical categorizer with a pretty visualization

syntactic.omershapira.com

61 pointsomershapira11 comments

It seems like your cluster quality will be sensitive to the words used to seed each cluster.

Why not use a standard word clustering algorithm like Brown clustering? http://acl.ldc.upenn.edu/J/J92/J92-4003.pdf

Percy Liang wrote a great implementation in C++ that you could plug into your visualization: http://cs.stanford.edu/~pliang/software/

Also of interest is that Brown clustering is hierarchical, so you can get coarse or fine-grained clustering.

[Aside: Here are some 2-d visualizations I made of word embeddings from a neural language model: http://metaoptimize.com/projects/wordreprs/ ]

I'll definitely look into the Brown clustering. I used this one with the hope of being able to eventually run spectral clustering on this (main problem: the KLIC defines a pre-metric, not a proper metric, so the order of operations is crucial).

How does the Brown clustering method guarantee less sensitivity to the seeding words?

Your visualization is interesting, I'd like to able to navigate it in order overcome most of the clutter. Originally I thought of mapping the clusters in 3d "clouds", but I think the dataset is too large for making a dimension-based visualization more than recreational - I mean, I'd probably be happier to read a cluster as a list.

Would love to see what clusters from PubMed would look like. Anyone planning to run this on it?

Agreed. Would love to see this too.

IIRC they have a pretty easy to use API but as far as data dumps (according to a quick search and http://www.nlm.nih.gov/bsd/sample_records_avail.html) it appears they only provide XML whereas the current code requires XML.

While it's true you'd get XML from NIH, as a first pass, just extracting the MEDLINE titles and abstracts into plaintext and then running this over them would be enough. You need to complete a license agreement (annoyingly) to download them. But there's no fee or anything.

> it appears they only provide XML whereas the current code requires XML

Is there a typo here, or am I just reading this wrong?

typo -- second instance of "XML" should be ".txt"

Anyone who wants to do that can contact me (contact form and email can be found there) and I'll happily publish the results.

First of all, great work and thanks for sharing!

I guess I know less about NLP and clustering than I thought, but what exactly does the visualization indicate?

On Iteration 1/3, when I click "husband" on the sidebar and "first" shows up...what does that mean? That that's the closest cluster by distance?

The visualization looks nice but the accompanying text doesn't shed much light...

There's a 'help' button on the top-right corner in case you get lost - but I guess I should curb my minimalism and make it larger.

The visualization is meant to visually explain the 'distance'. If the chosen word (top scope) falls down nicely on the target cluster (if any lit square on the top lights up on the bottom), then the word should be close. Note that it doesn't work the other way around (more about that in the 'The KLIC' section).

In the bottom of the screen there are two text displays - the left one shows the closest cluster with its contents and the right one is any cluster you choose. Note that the scopes change according to the selection.

need horizontal scrollbar

Reply on news.ycombinator.com