danielsf

nope. they are in there. if it's on the rap genius lyrics page, I included it.

the code is pretty straightforward, just chapter one of the NLTK book and a txt file.

nltk.org/book

I only have hip hop data, sadly.

I didn't expect wu-tang to be on the top since it would go across 11 different vocabs. I felt like the law of averages would basically play out.

OP here: many thanks!

op here: thanks yo!

OP here: the shakespeare thing is really just a hook, food for thought rather than an academic/cultural judgement.

I also had several suggestions to use shakespeare's sonnets rather than plays, which I should have done.

and yes, this is all just pinch of salt barbership discussion :)

OP here: the challenge is that most artists' best work is in their earlier years. I'd rather have Jay-z's first album than last, ya know?

here's what this looks like. ugly as sin as useless for comparing rappers.

http://www.mdaniels.com/vocab/scatter.png

love your other ideas – hopefully can do them later.

hm. it should be up. try it again.

OP here. Do I really need to provide all of this to satisfy the reader's ability to grasp the basic premise of the site? this isn't a thesis or academic pursuit, just comparing some rappers for fun.

I used plain NLTK token analysis on rap genius lyrics. in terms of several more data slices...I agree that there should be more cuts of the data, but you must understand the amount of time that it took me to put this together.

I never made the claim that vocab size was correlated with quality.

this is brilliant. OP here. We should work together.

since i use only rap genius data, i'm hoping the garbage all cancels out so that the rappers are still relatively in the right place.

I never made the claim that complexity is better/worse. I just wanted to communicate the data point, not imply 2pac>DMX.

I actually revised the conclusion to address this issue. Check it out on the site.

OP here. made up vocabulary is basically all of hip hop. look at any lyrics file and very little is "established", yaknowwhatimsayin?

this is an issue that i was hoping would cancelled out, given the fact that I use the same analysis for every rapper. In short, it's an exact unique word count, but it's relationally accurate.

OP here. vocab is non-linear:

mdaniels.com/vocab/scatter.png

so the average would change as your sample size grows.

35K was the threshold where things didn't get garbled (like the 100 word example that you mention).

The threshold is also impacted by who I include. If I went to 50K, I'd lose out on rappers like Drake.

author here: hit me up with questions you've got.

author here: oh man. flattered.

author here: thanks!

author here: the data is straight from RP, so I have been working with them :)

author here: indeed. thanks for adding this to the thread.

New York, NY – Product Development Lab, http://futureof.io/

---

We're hiring for a start-up in an end-up (a large global-scale company) – an opportunity for scrappy devs, designers, and product managers to work in a four-person, autonomous team that owns huge ideas from the start. You'll be building products in the payments/credit card space that will eventually scale to millions of users. The effort is inspired by Spotify's squad model and Lockheed Martin's Skunkworks.

Prove that the principles of lean-startups, when applied to a prestigious company with 50 million+ customers, can accelerate our pace of change. You’ll disrupt a company from the inside out, while producing ideas that have a meaningful effect on the daily lives of millions.

We're hiring: product managers, engineers, and UX/UI designers. H1B welcome.

More info at: http://futureof.io/payments

i just wanted to keep this safe for the non-hip-hop fans :)

Point taken though.

I'm going to kill the greyscale hover. It doesn't do anything meaningful and had no idea it changed the text.

What do you all think of this design? I used Medium, Exposure.so and Internet.org as inspiration.

thanks for adding!