Some Starting Points for Deep Learning and RNNs | Hacker News Reader

Some Starting Points for Deep Learning and RNNs

aistartups.org

149 pointsdenizyuret42 comments

Is AI right now useful as a way to provide value for small projects or startups? Any examples of that?

It feels like most examples of commercial AI usage are features (image search, automatic face tagging, etc.) that get added to existing products by large companies with access to large datasets and computing power.

I've been working on a side project to pulled news articles and gave me summaries of trends I might be interested in. I'm learning so the results are pretty poor... but so far I've spent less than $50/week running the project on AWS. You could go cheaper with spot instances/etc too.

Some big deep learning projects might be beyond small projects/startups, but I think AI is reasonable if you work methodically.

edit: $50, not $100.

Let me know how it works out if you want any beta testers, I have a slack/mumble integration I wrote that captures all the links we post, and I would love to be able to summarize what the link was about and offer that as well.

If you want to build a bot that does exactly what you are talking about, I just wrote an article [1] on how to do that! It uses an API with 1000 free calls per month, and is easy to host on your own servers or Heroku.

[1] https://medium.com/@samhavens/building-somerset-d518ba284c49...

Wow, super simple. I dont know what has been taking me so long to try this out, thanks a lot!

Took a look at the site and I signed up, some words of caution for anyone looking for the paid edition, but it works well for me:

* You cannot display this information publicly in the free tier, and you must display their logo for attribution on their page at a specific size they mention in their TOS. * They want your phone number for some reason.

* On the other hand, the api (AYLIEN https://developer.aylien.com ) either changed their limits or you might be just mistaken, it looks like the free tier is 1000 calls a day now, which is awesome!

There is more to AI than complicated neural networks. There are simple well-studied algorithms that deliver predictable results and work very well if you apply them to the right problem. Look into K-means clustering, linear regression and decision trees to get the taste.

Good introduction to the field: http://ocw.mit.edu/courses/electrical-engineering-and-comput...

Agreed. Deep learning receives all the hype these days (look at the machine learning subreddit), but in reality it provides very little practical value for the vast majority of data mining tasks that most folks will see. For that, standard methods like you mentioned (and don't forget SVM of course!) are still the go-to methods in my toolbox.

On a more practical side, there is a good AI book with a misleading name: Programming Collective Intelligence. Instead of going over some abstract algorithms, it poses a series of practical questions (recommend a movie, calculate optimal price, filter spam) and then shows (with source code!) how to solve them using common AI techniques.

Appreciate the recommendation (I wrote that)! Honestly the book is very old at this point, it may still be an approachable way to understand what's behind the algorithms but there are much better ways to write the code now.

I wore out my copy. Have you come across any books or course with the same approach but more updated code to recommend?

Are you speaking about just the code samples or commenting on the general approaches used in the book?

Is AI right now useful as a way to provide value for small projects or startups? Any examples of that?

I'm going to make the simplifying assumption of saying that "machine learning" is a subset of AI for our purposes, and that "machine learning" includes "simple" techniques like linear regression, logistic regression, k-means clustering, etc. I don't think any of that is terribly controversial, although I know a few people would quibble over it. Anyway...

Given that, I'd say the example is absolutely "yes". AI can help startups and small projects. There's actually a really nice example that Andrew Ng talks about in his Machine Learning course on Coursera. In that example, a t-shirt manufacturer is trying to figure out how to size their shirts. So they go out and take a bunch of height/weight measurements of prospective customers, and then use k-means to segment the data into clusters. Let's say they do 5 clusters, corresponding to extra small, small, medium, large, and extra large. Now they can look at the measurements in each cluster and figure out how to size their shirts.

More generally, anywhere that you have data, and you want to extract insights from that data, you likely have an application for some level of machine learning / AI. Of course it won't always (or even often, perhaps) be the case that you need a deep neural network, or anything like the cutting edge in academic research. But, then again, you might. :-)

that get added to existing products by large companies with access to large datasets and computing power.

Do you have an AWS account? If so, you have access to large datasets and computing power as well. Look at all the data that's "out there" in terms of Open Data, LinkedData, etc... As an exercise, browse around data.gov sometime, and look at things like the datasets the UN makes available, and the world bank data, etc., and see if you can think of a way to combine a few of those datasets and extract some meaningful insight from a combination nobody else has looked at before.

If you come up with something, it's not terribly hard/expensive to spin up a cluster using EC2 and run some analysis.

Yes. Language learning, for example. Speech recognition is being used for a lot of IoT stuff, home automation, and toys and the like. Lots of other really interesting niche applications.

We do research consulting in speech & language, and tons of small start-ups come to us with a variety of really interesting problems.

cobaltspeech.com

It's hard to see how speech recognition requires custom ML; isn't this offered as a service by various OSes and SaaSes? In addition, it's hard to see how this would be critical to the startup's success.

I'd expect trained classification to provide automated curation to be the best way for startups to take advantage in an existing project.

There are a lot of problems in speech besides transcription. I should have been clearer about what I meant.

Even for people that do want transcription, though, often you can handily beat the accuracy of off-the-shelf services by tuning the models to the domain.

I'm pasting some copy from our website in part because I can't talk too much about specifics, but here's some examples:

High quality speech to text transcription, including in the very difficult areas of conversational human-to-human speech: phone calls, voicemails, meetings, etc. Classification of speech & language: identifying gender, age, regional accents, level of education, etc. Voice front-ends for various applications and devices, including natural language interfaces: heads-up displays, robots, smartphone apps, etc. Speech synthesis (a.k.a. TTS) to generate high-quality synthesized speech from text. Analysis of audio: detecting different types of noise or speech: dog bark, shouting, gunshot, water running, cars, etc. Customization of speech recognition to processed signals, such as proprietary compression or noise reduction algorithms Fine-grained analysis of speech and language, for giving feedback in language pathology or language learning. Analysis of speech & language to detect various health conditions: stroke, dementia, depression, schizophrenia, etc. Information extraction from text or audio: phone numbers, dates, entities and relationships, etc. " [0]

[0] http://cobaltspeech.com/our-products.html

Sure, but best possible scenario you end up with a half-baked siri: interesting technology with gimmicky applications. Hardly a good use case to show case how ML can help a startup. The cost/benefit is huge for the general case using speech recognition.

> 'There are a lot of problems in speech besides transcription. '

I'm trying to be vague because I don't want to violate NDAs and it's a pain to think up examples I'm not currently working on, but there are a lot of applications of machine learning to the _speech signal_ (rather than to the problem of speech to text; there are many other things you can predict) that are super interesting. Some examples are listed in the text I copied.

> 'a half-baked siri' There are so many instances where speech-to-text is super useful for applications other than consumer facing CRUD apps. In a lot of varied industrial applications people are doing very cool things with speech; there are a lot of applications that need very targeted speech systems.

Interesting, sure. Useful or profitable is a separate point that needs a discrete argument. I'd argue it's still in the gimmick phase unless interface IS your product.

for example: https://wit.ai/

There are many not-incredibly-large advertising companies using machine learning to optimise (in many sense of the way it can be done) "what the do with (and how they do what they do) ads".

Disclaimer: temporary contractor for one of them.

I'm just now getting into neural networks and machine learning. I notice that there's a lot of research going into RNNs, CNNs, and deep learning.

So far, I've only really delved into NNs with one hidden layer, and I read that problems really only require one hidden layer. For what kinds of problems are multiple layers required, or CNNs, or RNNs? It seems that there's a lot of "cool factor" there, but I don't know what they bring to the table that's new.

Where I read about one-layer being "good enough" for most problems: http://stats.stackexchange.com/questions/181/how-to-choose-t...

Usually you need multiple layers when you train NN directly from raw features (like pixels or words) for perception-like problems. In that case you want to learn complicated multi-stage transformations to get useful high-level features.

If you work with already extracted features (like a classical ML pipeline where you can use SVMs or logistic regression as well), you may go with one-layer network. But deep learning really shines with perception-like problem.

Don't be fooled by the approximation theorem (you can approximate any function with only one hidden layer). It is beautiful in theory, but in practice it is like saying that you can write any program in brainfuck because it is turing-complete.

I found an analogy with electronic circuits useful. You can implement any bitwise function using a single layer, a lookup table. The problem with that is it requires as many logic gates as there are inputs, exponentially many. As a result, it's more efficient to design deeper circuits that compose multiple logic gates. The tradeoff is that it is slower to evaluate, since the signal has to pass through more logic gates.

That answer is from 2010, something to keep in mind. The field is evolving so rapidly you should try to find something more recent. And preferably from an expert in the field.

What are the classes of problems that can be solved by AI today? What are current challenges?

"Solved" is pretty broad (as is "AI") but deep learning, specifically, performs well (SotA) on a number of benchmarks in speech, language and image recognition. Some challenges immediately come to mind:

1. The pace of published research is pretty fast right now. This makes it difficult to know where the research fits in when solving problems. It'll probably take a few years before we know where to use many of the approaches published last year.

2. Iteration performance (trying many new things quickly) is improving with high-level frameworks but still lots of work to do here. Since we don't know where a lot of research fits it's not always apparent which methods work best (and "best" changes every quarter).

3. We're still missing theoretical foundations for much of deep learning. This is useful, not just for research, but to know what can and cannot work with current approaches.

4. Model architectures are still based largely on trial-and-error, intuition and search.

Generally speaking we have a handle on problems where you have lots of labeled data of the classes you want to identify (lots generally being quantified roughly by calculating N/(e^D) where N is number of entries and D is is the intrinsic dimensionality of the data. The larger that number is the more complex of a model you can train.).

Some hard long term challenges revolve around cases where you don't have a lot of unlabeled data or examples of classes you care about. There's also the technical challenges of training large scale models without obscene computational resources.

I also prefer to reserve the term AI for generalized AI, which we're still a ways off of, as opposed to modern classification problems etc. that I would call machine learning (though I know that nomenclature is uncommon).

EDIT: jimfleming makes a great point about theory as well - we could likely be much more efficient with better theory for deep neural nets.

N/e^D would be vanishingly small for most problems on which deep learning is used. Image recognition for example may involve more than 1000 pixels and e^1000 is a number that makes the number of the atoms in the universe look tiny.

The qualifier there is "intrinsic" dimensionality. There is smoothness to the image such that there is a lower dimensional embedding than 1m dimensions for a 1 megapixel image, particularly with any applied setting.

That said there is a bigger problem with those bounds because it doesn't incorporate model complexity. The VC dimension is much more insightful because the complexity of your model and the hypothesis space it represents is important for proper training. As an example, add a regularization term to your model and you're no longer doing anything like N/e^D. Convolutions, dropout, etc all prevent NN models from becoming too complex to train.

Do you know of a way to measure "intrinsic dimensionality" ?

In general it's an abstract concept like Kolmogrov complexity but there are some practical approaches.

People often try to intuit the intrinsic dimensionality of a dataset by using techniques like looking at singular values above some threshold or reconstruction error versus changing the output dimensionality of a dimensionality reduction/unsupervised technique like PCA, matrix factorization, or an autoencoder.

An info theory person might argue entropy and compression ratios are also insightful.

I'd also note that for the case of typical classification that there's an additional concept of "intrinsic boundary dimensionality" that should be low due to the presence of invariants like scale, rotation and translation which drastically lower the intrinsic boundary dimensionality.

Organization is a huge one that we see at Clarifai (I'm an engineer there). We do photo recognition based on neural networks and deep learning, and a lot of our customers have a TON of user-generated content that's hard for them to curate and manage, so we step in and our tech understands their photos and videos for them. clarifai.com

how relevant are hidden markov models now? Do they still have unique strengths that aren't covered by other, newer models?

Some state-of-the-art industrial speech recognition [0] is transitioning from HMM-DNN systems to "CTC" (connectionist temporal classification), i.e., basically LSTMs. Kaldi is working on "nnet3" which moves to CTC, as well.

Speech was one of the places where HMMs were _huge_, so that's kind of a big deal.

[0] http://googleresearch.blogspot.com/2015/09/google-voice-sear...

I'm kind of more interested in if HMM's are more viable as a general-purpose tools rather than their applicability in cutting-edge research; or if they should generally be avoided for common tasks unless the domain is simple enough.

I wonder how much of ANN popularity is due to the inherent strength of the model, and how much of it is due to large corporations funding/hiring people who happen to be interested in that particular flavor of machine learning.

HMMs are only a small subset of generative models that offers quite little expressiveness in exchange for efficient learning and inference.

Thanks

IMO, anything that be done with an HMM can now be done with an RNN. The only advantage that an HMM might have is that training it might be faster using cheaper computational resources. But if you have the $$$ to get yourself a GPU or two, this computational advantage disappears for HMMs.

It's almost sad – I always felt HMMs had a unique elegance that is lost on the brute-force NN approach.

HMMs also used to produce more meaningful intermediate results, although I guess these 'DeepDream' images set a new record in that regard :)

Not really. The first order Markov property is really naive when compared to an LSTM.

Reply on news.ycombinator.com