An Upgrade to SyntaxNet, New Models and a Parsing Competition
research.googleblog.com
research.googleblog.com
http://www.nvidia.com/object/gpu-accelerated-applications-te...
But it is a little bit out of date when it comes to version numbers.
If you're going to try TensorBox (https://github.com/TensorBox/TensorBox) it will get a bit harder still because of conflicts and build issues with specific versions of TensorFlow.
There has to be an easier way to distribute a package.
That said, all this is super interesting and Google really moved the needle by opensourcing TensorFlow and other ML packages.
I would really like to see some competition in GPU space. Nvidia monopolizing deeplearning rigs may not be the best thing for the future.
See http://reference.wolfram.com/language/guide/NeuralNetworks.h..., also look at Examples > Applications under http://reference.wolfram.com/language/ref/NetTrain.html for some worked examples. Fun example of live visualization during training (very easy to do, will get even easier in future versions): https://twitter.com/taliesinb/status/839013689613254656
The problem in the end is that the customizability of Mathematica ends right where things get interesting. If you want to show people cool little examples, Mathematica is clean and fast, but you can't build anything serious with it. And by "serious", I guess I mean something with few enough constraints to have an identity of its own, rather than being "a thing you can do with Mathematica."
Another limitation is the data input. Someone needs to rethink it.
I could be wrong. I actually want to be wrong, because of the simplicity and power of Mathematica in its scope.
Programming languages/platforms are network goods. IMHO, Mathematica has tried to swim against this fact and has failed.
Mathematica, which is a serious project, is largely written in Mathematica. Wolfram|Alpha, another large project, is built in Mathematica. Outside the company, https://emeraldcloudlab.com/ for example has built their platform on Mathematica.
> And by "serious", I guess I mean something with few enough constraints to have an identity of its own, rather than being "a thing you can do with Mathematica."
For research, a "thing you can do with Mathematica" is often what you want. But other than that you can put things you build in the cloud via APIFunctions (similar to AWS Lambda functions), or call out to them via wolframscript, or talk to kernels directly via MathLink or LibraryLink, or over sockets via ChannelListen.
> Another limitation is the data input. Someone needs to rethink it seriously.
That's vague, but I imagine you mean importers. Certain built-in importers aren't as good as they should be, like CSV and XLS are memory hogs and die on relatively small amounts of data (at least they used to, I haven't checked recently). The HDF5 importer is now pretty good, and for large datasets it's a good choice for scientific computing anyway.
> Programming languages/platforms are network goods. IMHO, Mathematica has tried to swim against this fact and has failed.
No arguments there. I think it would be great if we could open source at least parts of it, because no doubt new life would be breathed into cobwebby parts of the codebase and various pet peeves fixed. But Mathematica still dominates the computer algebra space despite being closed source, and probably will continue to do so for a while.
For research, closed source should be a dealbreaker.
Guess the only way is to upgrade to 11.1 and find out, but since you're apparently involved -- might as well ask...
The annoying thing for GPU training is handling the cudNN dependency, which Google's guides are annoyingly lacking.
I've found a nice workaround for that one that does not require registration with nvidia so you can automate it:
ML_REPO_PKG=nvidia-machine-learning-repo-ubuntu1404_4.0-2_amd64.deb
wget http://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1404/x86_64/${ML_REPO_PKG} -O /tmp/${ML_REPO_PKG}
sudo dpkg -i /tmp/${ML_REPO_PKG}
rm -f /tmp/${ML_REPO_PKG}
CUDA_REPO_PKG=cuda-repo-ubuntu1404_7.5-18_amd64.deb
wget http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1404/x86_64/${CUDA_REPO_PKG} -O /tmp/${CUDA_REPO_PKG}
sudo dpkg -i /tmp/${CUDA_REPO_PKG}
rm -f /tmp/${CUDA_REPO_PKG}
apt-get update
apt-get install libcudnn5-dev
apt-get install libcudnn5(these were the ones that worked for me).
16.04 is there as well:
http://developer.download.nvidia.com/compute/machine-learnin...
The standard build uses a "least common denominator" Intel instruction set (SSE4), but the odds are extremely high that the machine on which you're running tensorflow supports 4.2. Building from source allows you to use the most up-to-date instruction set (the default configuration script at https://www.tensorflow.org will do it automatically).
I've seen dramatic (>50%) reductions in processing time on test scripts by building from source. Note that those tests were built primarily for my own education, not for benchmarking. But the speedup was so dramatic that I couldn't help but notice and probe a little deeper. YMMV depending on the particular application, whether you are using GPU computing (I am not), etc.
TensorFlow is not meant for the CPU. If you want to do something serious you have to move to the GPU, which it is at least 8 times faster than your optimized CPU with an average GPU.
But it's a bit disappointing to hear that the build system is something of a nightmare, if I ever wanted to contribute myself. There's always plenty of things to help with, I don't care about the cutting edge of machine learning (I'm happy to submit docs, examples, etc)... Then again, the TF people can't just nerd around on their build system, for dorks like me to maybe write some patches every once in a while. Always great to make it easier, though.
And python has a huge problem with this anyway. (Or, to put it probably more accurately, I have a huge problem with python in this way, historically my python code has had a relatively short shelf life compared to my C code (or even my PHP code)).
Could you please share in more details the problems you encountered with Bazel?
If you have specific questions, I encourage you to post on StackOverflow with the bazel and tensorflow tags, both the Bazel and Tensorflow teams are monitoring and answering questions there.
I know... I am building it for TensorFlow Serving and it is pain in the ass. I will help but I have no idea how to do it, only find workarounds to compile specific commits.
See this issue:
https://github.com/TensorBox/TensorBox/issues/100
and
https://github.com/TensorBox/TensorBox/issues/102
So then we're full-circle and installing from pip which doesn't work :(
sigh.
Anyway, I'll get it to work, somehow.
For CuDNN I've found a good solution, see below.
Tensorflow itself worked ok once I figured out what all the dependencies were, even so I have not been able to get it to use CUDA yet (it only works with the CPU), which is strange because other CUDA stuff works fine.
-(Eng) We need to switch to this new NLP framework
- (VP) Ok, why? Which one is it?
- (Eng) Huh, it's called Parsey McParseface, developed by ...
- (VP) WTF? Don't waste my time with jokes, go build your own
- (Eng) But ...
- (VP) Meeting's over.
...and apparently will release a major version update today. Ouch.
>On the time-honoured benchmark for this task, Parsey McParseface achieves over 94% accuracy, at around 600 words per second. On the same task, spaCy achieves 92.4%, at around 15,000 words per second. The extra accuracy might not sound like much, but for applications, it's likely to be pretty significant.
If spaCy is able to increment the accuracy and maintain the large performance gap, it'll still be my go-to NLP framework!
I do wish SyntaxNet were a bit easier to use. A lot of people have asked for SyntaxNet as a backend for spaCy, and I'd love to be using it in a training ensemble. When I tried this last year, I had a lot of trouble getting it to work as a library. I spent days trying to pass it text in memory from Python, and it seemed like I would have to write a new C++ tensorflow op. Has anyone gotten this to work yet?
I wonder if the spaCy update will go deep :)
I've learned a lot while doing the neural network models, though. The 1.7 model takes advantage of this by having a more sophisticated optimizer. Specifically, I use an online L1 penalty and the Adam optimizer with averaged parameters. The L1 penalty allows control of size/accuracy trade-off.
This means we're finally shipping a small model: 50mb in total, compared to the current 1gb. The small model makes about 15-20% more errors.
[1] https://github.com/tensorflow/models/tree/master/syntaxnet
On HN the idea is to rewrite titles only to make them less misleading (or less baity). Please see https://news.ycombinator.com/newsguidelines.html.
https://github.com/tensorflow/models/blob/master/syntaxnet/g...
This new DRAGNN framework is what I thought the folks here would want to know. Perhaps I should have linked to the github page, rather than the blog announcement.
For the same reason, it probably doesn't make sense to change the current thread to point to that Github page now, since that would orphan the existing discussion.
The bit about guessing the part of speech, stem, etc. for previously unseen words should (I think) make it much more useful in contexts that succumb to neologizing, verbing nouns, nouning verbs, and so on (such as business writing, technical writing, academic papers, science fiction & fantasy, slang, etc.).
I wonder how well it would do at parsing something that seems deliberately impenetrable, like TimeCube rants, or postmodern literary criticism.
Other pathological cases are business books trying to coin a term or twist existing words into new meanings (eg. "cloud"), verbing nouns (incentivize), nouning verbs (likes, learnings), and so on.
Another HN thread has good tutorials for simple uses of Tensorflow: https://news.ycombinator.com/item?id=13464496
However, NNs are optimal for text/image data as they can learn the features. If your data features are already known, you don't necessarily need to use Tensorflow/Keras at all, and you'll have a easier time using conventional techniques like linear/logistic regression and xgboost.
I recommend http://course.fast.ai/ to learn more about the applications of neural networks and how to apply neural networks quickly through python.
It's a very interesting research carried out by Google's research team and I believe this will be especially beneficial for future speech translation algorithms that would bring us a whole new, fresh experience with the way we converse with Alexa, Google Home, Siri, and many more.
If you need to install TensorFlow onto your Windows 10 computer then here's a great guide which I have followed quiet a few times. :)
Alexa, turn the lights on in the kitchen.
Alexa, turn on the kitchen light.
Alexa, light up the kitchen.
Should all accomplish the same task using this framework.
Just simply treating the sentence as a bag of words and looking for "on" or "off" or "change" (and their synonyms) and the presence of known smart objects works extremely well. I could say "Hey Marvin, turn on the lights and TV", or "Hey Marvin, turn the lights and TV on", or even "Hey Marvin, on make lights and TV."
(It's named Marvin it after the android from The Hitchhiker's Guide, my eventual goal is to have it reply with snarky/depressed remarks).
Adding 30 seconds of "memory" of the last state requested also made it seem a million times smarter and turns requests into a conversation rather than a string of commands. If it finds a mentioned smart object with no state mentioned, it assume the previous one.
"Hey Marvin, turn on the lights." lights turn on "The TV too." tv turns on
The downside to this approach is I would be showing it off to friends, and it could mis trigger. "Marvin turn off the lights." lights turn off "That's so cool, so it controls your TV, too?" TV turns off But it was mostly not an issue in real usage.
Ultimately I've got the project on hold for now because I can't find a decent, non-commercial way of converting voice to text. I'd really rather not send my audio out to Amazon/Google/MS/IBM. Not just because of privacy, but cost and "coolness" factor (I want as much as possible processed locally and open-source).
CMUSphinx's detection was mostly very bad. I couldn't even do complex NLP if I wanted because it picks up broken/garbled sentences. I currently build a "most likely" sentence by looping through sphinx's 20 best interpretations of the sentence and grabbing all the words that are likely to be commands or smart objects. I tried setting up Kaldi, but didn't get it working after a weekend and haven't tried again since. I don't really know any other options to use aside from CMUSphone, Kaldi, or a butt SaaS.
I've wanted to add a text messaging UI layer to it. Maybe I'll use that as an excuse to try playing with ParseySaurus.
Same concern here... so my voice->text method is via android's google voice - forced to offline mode. The offline mode is surprisingly good.
Re mis triggers... I also have opencv running on the same android. It only activates the voice recognition when I am actually looking directly at the android device (an old phone).
I actually tried this at one point with a wall-mounted tablet before trying Sphinx. It is surprisingly good for offline, probably the best offline I've tried yet outside of dedicated software like Dragon. But it doesn't meet my open criteria, so I'm hoping to find something better.
I'll most likely give up on the requirements of it needing to be local and open, and use Sphinx for hotword detection to send the audio out to AWS for processing.
> Re mis triggers... I also have opencv running on the same android. It only activates the voice recognition when I am actually looking directly at the android device (an old phone).
That's an awesome idea :) I haven't gotten around to playing with anything vision based yet. But I've thought of 'simple' projects like that, which would add a lot to the perceived intelligence. Figuring out the number of people in a room would be another useful idea I think. The AI could enter a guest mode when there is more than 1 person in the room, or when it detects faces that aren't mine, or something similar.
With the leaps and bounds being made in ml these days it can't be long before magnitudes better open source voice recognition becomes available. I gave Sphinx a try but it was horribly disappointing.
For me, the combination of google voice (offline) and Ivona voice (Amy) is pretty damn good for my android/python/arduino based home AI.
May also be interpreted as:
Alexa, set fire to the kitchen
Alexa turns the gas stove directly to 'high', and waits.
Oh lord, is this the spark that lights the google skynet powder keg