Improving Stack Overflow jobs search with machine learning and R | Hacker News Reader

Improving Stack Overflow jobs search with machine learning and R

medium.com

154 pointsslashdotdash17 comments

I found it got off to a bit of a slow start, but was a fun, rewarding, and very real-world read.

In particular, I appreciated admissions like this:

> Writing our own genetic algorithm in C# was a bad idea. It took us weeks to implement, test, and optimize. Not to mention all the time spent waiting for results. There was a better solution available all along (the optim function in R). Because we didn’t do proper research, we overlooked it and lost time. Sigh.

... which far too few eng blogs overlook / fail to mention.

R is a very well supported language on Windows (since MS bought Revolution Analytics), very well integrated with SQL Server and Azure, and R Server solves a pain point that a lot of real-world users have. And of course it's open source.

Question to spark discussion and for me to fill potential gaps in my knowledge, not to criticize the article as I very much appreciate the transparency and build-up from the simple naive initial approach to the final approach used in production:

Is anyone else bothered by the claim that there "is a 100% chance that the new version is better than the current one" shown by using bootstrap? Maybe I've just never come across such a use of bootstrap through my encounters with statistics. I know it as a tool for resampling from a population to build up properties of your estimator (mean, variance, what have you) when all you have is a dataset and no clue about the actual distribution. When I saw bootstrap with that probabilistic claim, I thought the author would calculate a bootstrapped (100-x)% confidence interval for both the current and the new weights: and if the intervals didn't overlap with one another then you can claim with (100-x)% certainty that one is better than the other. But the author creates a new statistic that is a function of both datasets; Z_i = 1 if new is better than current on iteration i (on a random subset of data) else 0, and for all N=10000 iterations Z_i = 1. The chance/probabilistic claim made of new being better than current is based on the fact that no variation was seen on Z_i (I'm also kind of skeptical that out of so many iterations with random subsets that each time the new weights were better than the current). I think at most you can say that you simulated subsets of the data and 100% of the time new > current; the current claim leads me to believe there's inference that isn't there.

Maybe I should just ask one of my past stats profs. Open to someone enlightening me.

A quantile-based confidence interval from bootstrapping can yield a 100% confidence interval that does not contain 0, i.e., with 100% of cases positive/negative. But that does not (necessarily) mean that there is a 100% chance that the new version is better than the old one. Confidence intervals are not Bayesian credible intervals and cannot be treated as such. (That said, making some certain assumptions about the underlying model can in some times allow one to treat nonparametric bootstraps in such a way.)

Right. The author finds 100% of the time for his current dataset but makes a statement that implies some certainty or inference on future cases. Like taking 100 men, 100 women and finding that 100 randomly matched pairs had the man taller than the woman 100 times, and making the claim that there is a 100% chance that men are taller than women.

The more I type the more I realize how pedantic this is, but we're emphasized in stats to pay extra attention to the conclusions we draw from the data we analyze.

Great read. I loved the honest balance between engineering features and how much they ultimately ended up mattering to users. Oftentimes, the features we select can be quite arbitrary -- it's good to do gut checks by running real-time validation of results as often as possible. Fortunately, at Stack, you've got the userbase to do just that :)

> Genetic algorithm running on 56-core machine

Wow, just... wow. I wonder why they didn't utilize (multiple) GPUs instead? I would guess it would be far more efficient in all aspects. Especially now that there is TensorFlow & co.

$150k on GPGPUs seems a bit steep if you're going to throw that approach away anyway...

Why $150k? Ordinary GPU (<$1k) would probably be equivalent to 56 CPU cores, if not faster.

I imagine the reason for using CPU was that they would throw it out anyway, so there was no need to make an ideal solution - and they already had those 56 CPU cores.

Or you can spin an instance on AWS of your choosing?

Yes, of course. I seem to remember that their data is still in their own datacenter though, so they might not be comfortable with using Amazon for this.

That was just the price of some nVidia kit I was looking at, roughly.

I had no idea they have this, but it would probably be an overkill for SO. An ordinary (nVidia) GPU which starts at $150 is roughly >10 times faster at NN problems than 4 core CPU.

This: https://www.nvidia.com/en-us/data-center/dgx-1/

That's precisely why, indeed. This 56-core machine is what was at hand at that moment. Genetic algorithms is something we tried, but we didn't have enough confidence in it to invest too much money and effort. Which was a good call, since we soon realized there was a much better approach.

At the time the algorithm was running on one of the servers in the data centre and until recently we didn't have anything in the DC kitted out with GPUs (see http://blog.marcgravell.com/2016/05/how-i-found-cuda-or-rewr... for the reason behind that).

I'm not sure if we're planning to use GPUs for future iterations of algorithm testing; the GPUs are installed in hardware serving production traffic so it's unlikely to be used for experimental stuff like this. That said, a bunch of us just got Dell XPS-15s with the NVIDIA GTX 1050 so we can probably have a go on our local machines to get a feel for performance characteristics... watch this space, I guess!

Because they built it in C# by hand and had already spent too long on the wrong method, when R can do it 70 times faster?

It's not a neural network, so everything would be hand coded TF operations. It's unclear how that would be quicker than gradient decent.

I'd be more interested to know why they didn't try the learning-to-rank implementations in XGB or LightGBM?

Reply on news.ycombinator.com