Hacker News Reader

Comment by cageface | Hacker News Reader

Are support vector machines still used much or have they been supplanted by deep learning methods?

They’re really useful (and arguably state of the art) in situations with small amounts of data. Deep learning is really hard to productionize, so classical techniques like SVMs and random forests are widely used in production. Deep learning is too, but not as much as you’d think.

I graduated in 2006 (undergrad CS degree), and at the time, we were told SVMs were a lot more practical than something like neural nets. Neural nets were framed as "we will teach you this thing because it's fun to code back-prop and it kind of works in a way we think your brain does too, but no one really uses them in real-life, except for classifying digits".

Funny how times have changed.

Funny how that happens. In 2006 I was taken a grad-level Intro to ML course. After the first (and only) lecture on neural nets, I asked the prof on recommendations for learning more - since I had mostly a cogsci background, I was pretty interested because of the "neural" part. The prof essentially said the same as yours (I don't blame him -- it was a common sentiment of the time). And of course, these days he's doing deep learning!

Even in 2014 (grad cs degree), my data mining professor said SVM were advantageous over neural nets due to local maxima problem, so we never learned neural nets in class.

That's a bit weird. Sure it's an advantage if you can optimize the object function more easily. But the end goal is generalization, i.e. performance on new data. The objective function is only a proxy for that.

I still use them. I had to make a few binary and multi class classification models in NLP and while Transformers models gave nice results, it was impossible for me to use them long term.

Why? Because data were confidential, I could not use the Cloud to get a nice GPU, and I needed almost one model per enterprise. Training DistilBert for 3 epochs was like 12 hours and it was a domain where data/concept drift happen often which means re-training was mandatory on weekly basis, at least.

After some feature engineering, I was able to get almost the same performance (<1% difference) with a model that was able to train in less than 5 minutes and could be use in production easily.

I love the fact that SVM, at least in scikit-learn, have a built-in early stopping mechanism. Really handy.

For large data yes. For small data sets feature engineering is often more important than the machine learning model used. (Kaggle contests are frequently are won by Random Forrests + Feature Engineering)

Another important thing to consider is that a lot of the actual work in setting up a machine learning pipeline has nothing to do with machine learning (making sure that data is clean, exceptions are handled, etc). On a small data set an SVM can achieve good performance out of the box. Starting with an SVM can be a really helpful in validating the basic approach and understand the nature of the problem before you sink in a bunch to time/money collecting data and training a complicated model.

I used at work as part of a NLP system to extract valid relations between named entities on context of politics. Since I didn't have labeled data before, I annotated some hundreds by myself, and because was not much label data SVM outperforms other models (compared with ANN, Gaussian Process, KNN, Naive Bayes, and others that I forgot). The best kernel for this approach was a Linear one.

The system has a compound set of machine learning models and parsers to finally extract from the government official public news (http://in.gov.br) documents with the following structured info:

- who was/will hired and fired (PERSON entity)

- which job role it will/did have. (JOB entity)

- when will happens¹ (DATE entity)

Each entity is extracted individually using a custom trained NER and each sentence is passed to the Relation Extraction system, which is built using SVM. Features are concatenated word vectors² compound by the slices of the text in the form (entity1, entity2, before, between, after).

The system is being alive for almost two years. It produces great results. Just did need retrain the entity recognizer twice in all that time (built using spacy which uses a averaged perceptron).

The SVM part (Relation Extraction) was not retrained since the first day deployed and it still works gracefully :D

¹this info is on the text as natural lang, sometimes is different from the post date ²gensim.Word2Vec custom model trained on this corpus.

I would like to under how you solved your feature engineering issues?

In general I used a custom word vector model with a fixed dimension of n=100. As you should now, this feature transformer only maps word to vectors, but I have a more complex structure than just words to work on. The model works sentence wise and preparsing of text is made, let's say I have the following sentence:

"Hire Bill Gates as Front-End Developer at 05/05/2020"

Suppose the NER extracts the following entities from it:

- Bill Gates (PERSON)

- Front-End Developer (JOB)

- 05/05/2020 (DATE)

In my domain problem, I need the relation PERSON-JOB-DATE, which can be decomposed in two binary relations: PERSON-JOB and JOB-DATE. Each binary relation it's a model by itself with the following class outcomes: invalid, hiring, firing. If two binary relations has the same job and outcome classes, I build the triple PERSON-JOB-DATE.

At feature engineering level, which is you asked for, I build a tuple of local attention from entities perspective based on slices of the text: (entity1, entity2, before, between, after). Each part it's transformed into a vector by using average word vector and finally each part it's concatenated in a final vector that will be used by SVM. Since word vector dimension I choose in my experiments was n=100, my model will have n=500 features.

An example about the slices for PERSON-JOB it will be:

("Bill Gates", "Front-End Developer", "Hire", "as", "at 05/05/2020")

1. Each element of the tuple is tokenized

2. With tokens available, use word vector model to transform each one.

3. For each element of the tuple take the average of the vectors.

4. Concatenate each averaged vector into a final vector.

Using that structure the model is biased strongly by how the sentence is written with the words around of the entities. For my domain where the sentences are very regular it worked well. I am not sure if will work in more general domain, like social media.

I hope this clarify your question in some level.

Thanks so much

Yeah they're still used. When approaching a new problem I reach for linear methods first if it seems like they could work. DNNs are amazing, but can be very heavy handed and use significantly more resources, which isn't desirable unless needed or one is working with a complex problem.

I've even used DNNs when I couldn't make SVMs work, but someone more experienced came along and showed me a kernel method that for the trick with an SVM.

In short, it's a great tool if it works.

Reply on news.ycombinator.com