Things I Learned Building The Most Powerful Language Processing Engine
xyhd.tv
xyhd.tv
Linguists don’t know squat about grammar in modern times. Everything is a verb these days. I Google things. I FedEx things. My game gets nerfed.
Every problem he talks about is well-known. Not to say that they aren't still problems- they are problems, and hard ones, that linguists (computational and otherwise) don't have good standard solutions for. Thing is, most of the time, you can get away with not bothering to solve the real problem. Most of the time, you can pretend that white-space separates words and periods separated sentences and that 8 parts of speech is good enough, and it will be good enough. And thus the incentive to spend lots of time and research money on solving all of the big problems in natural language processing is reduced.
Except when it isn't good enough, and we wonder "why the heck hasn't this been solved yet? Doesn't anybody realize that this library is totally broken?" But yeah, we know that it's broken, we know what the problems are, they're just really frickin' hard.
It is like swum. It is a real word, and I have swum many times in my pool, but it is not how humans write. More of practical linguistics is about how people use words, not how they are supposed to use words.
You could perhaps instead mean "standard" instead of "right" - that they're describing a standard dialect, which doesn't meet reality. That could be true, I'm sure it is in places.
Btw, I'd be interested to know how many parts of speech the CGEL describes. Anyone got easy access?
Man! How do you come up with these "facts"?
> I linked to the quora as just one example.
So, Ɐx(CommentsOnQuora(x) -> Linguist(x))?
And by the way, linguists are trained to think like this: http://www.youtube.com/watch?v=8nYmWt1J4Lg. If you want a wider commentary, read http://languagelog.ldc.upenn.edu/nll/?cat=5
You guys probably do good work. Making wild claims reflects badly on the company; all the more so if the CxO does it.
-The copy throughout the site does not inspire confidence. Your point that many texts are written poorly is valid. That does not require you to also write poorly. I've seen far too many sentence fragments on your site. You site also includes some embarrassing misuses of words, like "Some may believe using heuristic science in language analysis infers it is a learning system." These incidents discredit your claims about your technology.
-I am having difficulty finding specific information about how your technology works.
-The people responsible for your graphics, visual design, and the video about the summarization app should not have those responsibilities. Summly had nice a aesthetic; your aesthetic is jeopardizing your credibility.
I apologize if my comments sound hostile. When you make claims as bold as yours you should prepare for scrutiny. The problems you are addressing are interesting; I wish you good luck.
SRI is a respected brand with a great reputation and history. I wouldn't be so dismissive of marketing since, as you mention, you have recently exited stealth mode.
I am trying to be helpful when I say that for a company that produces language processing technology, your customer-facing content shows a strange disregard for language. If your writing does not demonstrate that you have mastered English grammar, why should I believe that your software has?
All cards on the table, there is a lot of discussion internally about who we are writing for. The result is we get a mish-mash of copy from Engineering and Marketing.
The PDF I linked to is much better because those were targeted to Engineering managers. The website still doesn't know who it is targeting. http://www.tldrstuff.com knows its audience it sucks much less.
Nope, gavinh is judging whether the product is even worth looking at by its marketing.
I was all set to hate this post, but I found I ended up largely agreeing with it, especially the quoted bit above. I remember working on a very focused NLP tool a few years ago and needed a comprehensive English Lexicon. No problem, I'll just scrape WordNet or similar, not even close. Then you start dealing with stemming and conjugations and such and realize that almost all of the algorithms for dealing with this kind of thing would barely even be hacks in software terms, yet there they sit, regurgitated in countless libraries, generating garbage stems all over the place. It ends up just being easier collecting all the stemmed forms as well and just building some smart in-memory indexes and data structures for searching millions of words.
Vector space models? Why do they work? Nobody really seems to know! Just jam all your words into some matrixes and run some simple calculations and voila you get something seems to kind of like it sorta works some of the time.
Sentence tokenization is stupid hard, but shouldn't be, parantheses have all kinds of different meanings, commas are a mess...English is stupid.
The worst bit though really is that most of the research->turned into software assumes astonishingly brittle models about the language which almost never seems to describe any actual usage of the language which always means very frustrating almost right results out of NLP systems. My previous sentence, for example, would cause most NLP software to blow a gasket.
Typos are really hard for rules systems.
PS Glad you didn't hate it.
Try one of these: http://www.undocs.org/A/Res/66/2 (start from finding the end of the first sentence).
Or, you can have a go at a whole corpus of these broken at paragraph level at http://www.uncorpora.org/
I am curious about one thing. Are you just handling white spaced words, or do you handle lexical items as well such like polywords (inside out), phrasal verbs (put up with, beat up), idioms, etc.?
Idioms we haven't addressed much. Fortunately in news they come up less than other things. Sarcasm we deal with in most cases, but not all. There are hints that things are sarcastic, but if they are written like "the Onion" and they don't change tone because they are really a parody rather than sarcasm we can't tell that.
I like to say that if your 5 year old would figure it out we will figure it out. If your 5 year old won't probably we won't either. But this is a huge leap forward since the competition is about the level where your dog would understand.
>I am the CTO of Stremor, we make TLDR Reader. Sentence objects Sentenceobjects represent information extracted from a single sentence within a document. Attributes and methods available on Sentenceobjects: ∗ text: The raw text of the sentence as a string. ∗ names: A list of all names detected in the sentence.
Yeah, this is similar to picking at Google Translate not translating Google into the same text as they use on their other-language homepages. I'm honestly not complaining, I just found it funny :) To be fair, it does a significantly better job at other blocks of text I've thrown at it. Not great, but surprisingly good - I'll have to prod it more.
The TLDR version is optimized for news at up to 5000 words. We have some other stuff for Fiction but it isn't public, and a version that is specific to politics.
Yes, language processing is hard. There are two challenges here:
1) Understanding the ambiguities of language, when every word in the sentence can be 2+ parts of speech.
2) Making it fast, and making it fit within the relatively constrained RAM limits of App Engine instances.
We're wrestling with both while we greatly expand on what Liquid Helium can do. It's not easy, but some of the things we're able to do with it are pretty magical.
We made this choice because with a small team it gave us the most freedom to focus on the code, rather than the scalability and the infrastructure. We don't manage servers, or routers, or load balancers.
http://googleappengine.blogspot.com/2013/02/app-engine-175-r...
There are other options outside of app engine that could help you achieve the same thing. I outsource most of my processing to picloud (specifically because I need more ram than I have).
I don't work in your field but I think it's a really interesting problem you're working on.
Also the best targets for acquisition put Google high on the list so building things the way they would makes sense.
Can is really annoying. Can is a verb. Can is a Noun. It is part of a Noun Adjunct.
The rules to determine "The chicken soup can help when you are sick" and "The chicken soup can landed on my foot" and "We can peaches to eat later" is very hard. More so if you put them in a complex compound sentence.
Yes it does most Germanic and Romance languages.
Yes it does domain independent named entities with a a higher score than anything else on the planet. ALL English classes. Medical, Dental, Animal. (that doesn't include Latin uses of animal names) Technical.
As I said we are just stepping out of stealth. I linked a PDF in the comments here.
My question about speed referred to syntactic parsing specifically. I am sure you can do entropy scoring faster than 200ms per sentence, but unless you have access to parses you are unlikely to be able to do more than purely extractive summarization. That's what Summly does, and every other summarizer on the planet as well. (Except perhaps Columbia's Newsblaster, but that's a bit of a different story).
I don't have time to do conference papers.
Our pipeline requires almost every one of our capabilities in order to do TLDR.
We have to grab the page. We have to separate the content from the theme. We have to convert the HTML to a not HTML "thing" that lets us work on the text but maintain the HTML. Then we have to Disambiguate/Segment the sentences. Then we have to analyze the type of content to pick how we are going to summarize it, which requires all the noun, and stemming and keyword analysis, then we have to rank the sentences in importance based on concepts and causation, and readability, and emotion. Then we have to put all the HTML back, and present it to the user.
We set the goal that Tom Sawyer can't take more than 45 seconds to run.
"How to Get Startup Ideas
[1] [2] [3] You want to know how to paint a perfect painting? It's easy. Make yourself perfect and then just paint naturally. Live in the future, then build what's missing. [4] [5] [6] [7] Live in the future and build what seems interesting. [8] [9] [10] 10 [11] 11 [12] 12 [13] 13 [14] 14 [15] 15 [16] 16 [17] 17"
doesn't seem to work at all...
Also Paul's writing is pretty poor. The ideas are good, but he jumps around and uses short sentences with far too many pronouns.
Garbage in Garbage out.
Here is the 25% version, which I think is Readable:
The way to get startup ideas is not to try to think of startup ideas. And yet by far the most common mistake startups make is to solve problems no one has.
I made it myself. But galleries didn't want to be online. Because I didn't pay attention to users. Because they begin by trying to think of startup ideas. That m.o. is doubly dangerous: it doesn't merely yield few good ideas; it yields bad ideas that sound plausible enough to fool you into working on them.
At YC we call these "made-up" or "sitcom" startup ideas. But coming up with good startup ideas is hard.
For example, a social network for pet owners. Millions of people have pets. Choose the latter. Not all ideas of that type are good startup ideas, but nearly all good startup ideas are of that type.
Made-up startup ideas are usually of the first type.
Nearly all good startup ideas are of the second type. If you can't answer that, the idea is probably bad. But you almost always do get it.
But while demand shaped like a well is almost a necessary condition for a good startup idea, it's not a sufficient one. If Mark Zuckerberg had built something that could only ever have appealed to Harvard students, it would not have been a good startup idea. Facebook was a good idea because it started with a small market there was a fast path out of. So you spread rapidly through all the colleges. Often you can't.
> The biggest thing I learned. The thing I also hope my team has learned. Everyone else has hit the limits of what they can do because they weren’t willing to burn it all to the ground and start over. We start over a lot. We code for 3 days and then decide this won’t work, and we do it over again. We take the lessons we learned but little of the code. The second, or thrid time we do it right based on what we now know.
I am a big fan of rewrites, and am sad to see how much of the software engineering world has centered around the Spolskyesque idea of rewriting being a horrible waste of time. Truth is, 99.9...% of software just sucks. The more and more infrastructure we build up around it, the more constrained we become. Just look at the most common languages/tools/technologies we use on a daily basis. C++/Java/JS/Python? These are just awful. Hell, I'm even like Python compared to most languages, but there's still so much historical cruft, and it's ridiculously slow to boot.
I think this is mostly because people just suck at writing software in general. Rewriting in the industry usually doesn't buy you anything since it's the same people writing it, and they're still constrained by all the other software that they interact with. But if everyone was more willing to rewrite, focusing on code quality, I believe our tools would be less intertwined, we could achieve much faster rates of progress, and the life of a software engineer would be a lot more tolerable.
I think for us the biggest thing is that often we don't know what approach will work best until we try, and often we have to balance more than just readability or performance. Memory usage issues meant that we prototyped one way for getting all the words in to memory, proved that we had enough information about the words for 99% of what we were going to do, and that we could get the other 1% later, and then re-wrote with a different "loader" and different information about the words loaded.
I like functional programming but it made sense to make parts of the code object oriented not just for readability or ease of programming, but because it was more performant.
So our rewrites are partly for performance, partly because the end goal moved slightly, and partly for maintainability. As we stack on new features often the way we should do things changes drastically.
Absolutely--and I think pretty much all software is like this. Engineering/optimization is, in a vague sense, all about how much shit you can throw at a wall and how well you can tell what sticks. The faster/easier/more accurately you can do this, the better. If you're going to be bound to the first pile of shit you throw due to business constraints or whatever, you're not likely to end up with something very nice.
I'm hoping this was a subtle and brilliant joke and not just a typo.
As a developer I feel your pain. Balancing our building, business concerns, and support for an API is hard. We are a small company and are doing our best. But if you do email sales we will make sure to get you access to the API as soon as it is available.
I know it sounds like a bold statement, but I believe it to be true. Doing things right required we build our own tools, and not rely on libraries from third parties. I think we benefitted a lot from that philosophy.
Like the general approach, which looks very promising.
tl;dr
The "short" 25% view is pretty good on this. The summary doesn't read as well because it picks up the ReadMe because I didn't mark it as code snippet because I'm sucky at WordPress editing.
I'm confident few of the Engineers for F1 racing are spectacular drivers.
I know my PE teacher couldn't touch his toes.
Engineering race cars well is distinct from driving race cars well.
If your PE teacher were capable of touching his toes, he would discredit himself by choosing not to.
We also have tech that could vastly improve book and movie reccomendations based on the content and themes of books, not just "did someone like you like this" kinds of systems.
It is too bad because apparently we wouldn't need to study language and cognition any more if we had access to your work. I mean, why bother with neurolinguistics or anything having the stink of biology or psychology? Brain mapping projects and experimentation have nothing to tell us now that we know that there are 3.2 million words in English. That has been the burning question after all.
You've made some progress working on the most widely and deeply studied language in the world with a richness of written data available that is unprecedented. But the linguist working to record, preserve and study a small Amazonian language that works nothing like English is probably just an idiot because they haven't figured out how language works yet.
Do you even know what linguists are?
Will I ever share? Possibly. But like Google it will likely be when I have a competitive advantage. I don't not share because I'm mean. I keep things close because as someone on the thread mentioned that they are competing if I published all the information you mention, then Google would build what I have built and we'd be out of business.
Those brain mapping projects aren't done purely for the good of mankind they are trying to make money and get funding. My goals are the same.
It's hard to believe those libraries suck in their entirety (I use NLTK to do automatic categorization and it works reasonably well). Just listing areas of improvements (and where to begin) would go a long way.
Of course it would take some of your time, but it seems you have done this work already, so all you need is to write it down. And I think it would buy you a lot of goodwill, whereas simply bragging that your technology is "light years ahead" of everyone else simply raises expectations... that your product'd better meet! ;-)