mrdrozdov

An excellent read! I had the same thought. It's almost negligence not to mention it.

Thanks for the clarification! I interpreted Rails as 'Rails' not 'Rail's' and 'template' as a verb (sorry for the specifics, but I'm at a linguistics conference after all). I've been using Pyramid/Pylons recently and particularly happy with the experience so far. Not sure why it's not more popular, although it's also not particularly a bad thing this is the case.

Can you explain the arguments against Pyramid/Pylons?

Do you think the community has professional writers or only writing enthusiasts from the tech world? :)

At quick glance, this work seems related to Information Maximization like is done in the papers for InfoGAN, VIME, and Intrinsic Motivation (for automatic goal-setting in RL).

Received: 20 August 2015

Accepted: 05 February 2016

Published online: 08 March 2016

Views: 19,562

Citations: 13

I think LSTMs are far from being replaced. They work too well and are too simple to use.

I'd put a little more faith in LSTMs. There's a lot less evidence for what they can't do than what they can. With enough fiddling, can get LSTMs to work for most tasks.

I'm not sure they're quite as complicated as you're making them out to be. If they are, then try a GRU instead ;)

This sounds like an argument that you can make for many fields. In data science, certainly you can cover topics like clustering and PCA, which are reasonably advanced, but straightforward to teach because they can be visualized very clearly.

I think it's worth considering data science as a discipline to teach youngsters rather than computer science. It's quite visual at times, requiring less abstract thinking, although does depend on important algorithmic principles to process large amounts of data. Besides, data science is readily applicable to many other fields including Econ, languages, history, certainly math, but even art.

The research in CS is very active. Take a look at arXiv/cs to get a sense:

https://arxiv.org/list/cs/recent

I would say Machine Learning is particularly "hot" right now, with the anecdotal evidence being that many ML conferences (like NIPS, ICML, CVPR, etc) have been experiencing better than linear growth.

I love this provocative title. For clarification, the author's argument is related to a person's ability to follow a path determined by intuition in the absence of proof. The examples given are of scientists who found ultimate success even though they initially only had a kernel of an idea, in contrast to someone who would have found success but at the onset had a clear vision of how they might get there. I agree wholeheartedly with this sentiment, and early in graduate school my cryptography professor gave a concrete example: Diffie and Hellman knew there should be some way to securely exchange a key long before publishing their now famous exchange protocol (I'm paraphrasing heavily what the professor said, but hopefully this illustrates the point).

The "tension between creativity and accuracy" came up in a slightly different context at a party this past weekend. I was discussing with some other recent grads the flawed method of peer review in research journals, especially as it relates to reproducing results of well known papers. The crux of the discussion was whether the progress of science has benefited from the lack of strict guidelines for reproducing results and whether some papers are still important because they described an interesting idea although were later found to be irreproducible or based on false data. I would say that definitely the field has progressed, although there is certainly an intrinsic (and expensive) cost in having papers that are irreproducible and scientists should certainly strive to make their results as easily reproducible as possible. In the absence of necessity, I'd go further to say that having easily reproducible research is so valuable that the contrary is simply not worth it.

Unrelated to "value of research", there is a notion of fairness that should be considered. If being flexible on accuracy is a "competitive advantage" of sorts in research, then it's important that this is made obvious. When the principle of accuracy is implied it becomes a hurdle to newcomers who would follow unnecessarily difficult path given they have no way of knowing a priori that being inaccurate (even slightly) is allowed, possible, or beneficial.

I received a handful of emails specifically asking for the list of schools I was referring to. Here is a list of 31. There are certainly schools that I have missed, but I think that any of these programs offer a strong curriculum and community that would prepare a student for a career working in data science.

https://gist.github.com/mrdrozdov/bd523362210892e6a4e11eb575...

I think there are good points here!

> Spinning up some K80s on Azure, installing TF/CUDA/OpenCL

I think a single k80 instance is roughly ~$1/hr. If you had an experiment running 24hrs a day for a year, you'd spend a little over $8.5k. You can build an equivalent desktop machine for less than $2k [1], which might be slightly more convenient (once it's built), although I haven't really factored in energy costs.

> That's how you get into it as a garden variety dev.

Btw, you don't really need a GPU to start learning about deep learning. You can train a SotA modal on MNIST using Caffe I think in roughly 10m on CPU (maybe 1m on GPU). You can also train a reasonable sentiment classifier or natural language inference classifier in less than an hour on CPU. My perception is that these types of tasks are really solid for someone who is beginning to learn about machine learning or deep learning, as they'll provide a playground to mess around with different optimization techniques (SGD v. SGD+Momentum vs. Adam vs. etc), regularization (L1, L2, dropout, batch norm, etc), data augmentation, error analysis, and so on. If you do an ML interview for an entry level position, chances are these are the types things they will ask about.

I guess deploying ML solutions for a company you are working at is a different story.

> Otherwise, go for the PhD if you want to actually make new stuff.

There's some truth to this! PhD (like a Master's) probably doesn't make sense most of the time as a dollar-efficient career move. Rather, it's something you should pursue if you find being in an academic environment personally satisfying. You definitely don't need to be in a PhD program to work on new stuff (although it might make things easier because you will hopefully be surrounded by lots of fresh ideas). I've heard about people in bootcamps working on novel research. Now that so many powerful tools are open source and easy to use (Pytorch, Tensorflow, etc.), it's pretty easy for anyone to put together a novel model.

[1] pjreddie.com/darknet/hardware-guide/

> And congratulations for you achievement and your bravery to quit the job!

Thanks!

I think this sentiment is roughly correct. It's basically the same for programmer as there are many jobs that involve programming. Note that data science doesn't necessarily mean machine learning, and there are lots of data scientists who do powerful work with stats that doesn't involve anything that looks like ml.

The jobs for people working in machine learning tend to fall into: software engineer (machine learning), research engineer, and research scientist.

Yann LeCun has an excellent Quora Q&A where he delves into the differences of the roles. I do not have the link on hand but it is probably easy to find.

I believe it is similar compensation as a software engineer with a specialty (front end, data engineering, databases, etc.). I would not recommend this career path to anyone that is in it for the money, but rather to do it because they find machine learning personally fulfilling. These other specialties can be personally fulfilling as well, and many of my close friends have a specialty that is not machine learning and are very happy with their career.

I did my undergrad at U. of Michigan, and my master's at NYU. I would highly recommend both programs, although there are many great ones out there. I probably have a list of ~30 schools that I think would be excellent for a master's in CS with a focus on machine learning (although it is worth considering a master's in Data Science, as this makes sense if ML is your primary interest).

I am happy to give more targeted advice on grad school. Please send me an email at andrew [at] mrdrozdov.com.

My undergrad was in computer science, although I would say this wasn't the case for most of my classmates. There were a lot of students from other engineering fields (like Mech E or civil), Math, Stats, physics, bio, and business.

Stanford parser is very good for preprocessing data. Things like part of speech tagging, named entity recognition, and dependency parsing. If you want to do something fun and interesting with your data, you will probably need to implement it yourself. Note, there are lots of other go-to tools nowadays besides the Stanford parser. Things like GloVe embeddings, open source translation systems (harvardseq2seq, open source sentence encoders (facebook fasttext) are probably necessary in many NLP pipelines.

When things "just work" with off the shelf tools then you probably don't need the researcher (although sometimes you will need them to just find the right solution/tool). When things don't work, you will need them. I guess this can be said about many fields though? (Databases, front end development, etc)

In Feb 2015 I quit my job as a front end developer to learn more about machine learning.

First, I went through the Recurse Center, which is a 3 month program sort of like a writing retreat for programmers. I learned a lot about Python and AWS in that time, and got an internship as a data engineer.

In that Fall, I started a computer science master's. I've taken mostly courses in machine learning including: Machine Learning Theory, Deep Learning, Probabilistic Graphical Models, NLP, and GPUs. I've collaborated with two professors on research papers, which has definitely been the highlight of my degree although I definitely think the courses were necessary as I continue to use the information that was covered.

Finally, I'll be starting this summer as a research engineer doing deep learning! This process took me 2.5 years, but I feel very prepared for my new role. It probably is possible to do this faster by joining a program like Metis or Insight, which prepare you for data science like jobs w/in 3 months. I would say that approach is slightly more challenging / high risk. If you really want to go into machine learning, I'd say doing the degree is a more surefire approach, granted it's more expensive in time and money.

In this work Convolution Neural Nets (spatial models that have a weakly ordered context, as opposed to Recurrent Neural Nets which are sequential models that have a strongly ordered context) are demonstrated here to achieve State of the Art results in Machine Translation.

It seems the combination of gated linear units / residual connections / attention was the key to bringing this architecture to State of the Art.

It's worth noting that previously the QRNN and ByteNet architectures have used Convolutional Neural Nets for machine translation also. IIRC, those models performed well on small tasks but were not able to best SotA performance on larger benchmark tasks.

I believe it is almost always more desirable to encode a sequence using a CNN if possible as many operations are embarrassingly parallel!

The bleu scores in this work were the following:

Task (previous baseline): new baseline

WMT’16 English-Romanian (28.1): 29.88 WMT’14 English-German (24.61): 25.16 WMT’14 English-French (39.92): 40.46

Is it so hard to publish at arxiv, share the paper on twitter, and submit a link to HN? At a minimum, arxiv is better than a blog post because there's some sense of versioning.

For anyone who is curious, some conferences share the peer review comments (even though they do not share the identities of author or commenter). Here are the comments from iclr2017 (https://openreview.net/group?id=ICLR.cc/2017/conference) and here are comments on Ian Goodfellow's GAN paper by Jürgen Schmidhuber and others (https://media.nips.cc/nipsbooks/nipspapers/paper_files/nips2...).

This is a really clever data collection technique. ;)

From TNW. Seems relevant:

Update 2, April 11, 11:15ET: Some readers have suggested that the allegedly deleted tweets might have initially appeared as replies to now-deleted tweets, which would explain why they are missing. However, numerous users contacted by TNW rejected this premise, claiming the missing posts were standard tweets.

I am curious about DeepMind/Brain's role in the curating search results. Would you mind explaining more?

Also, I am not sure I understand your reference to Youtube. Is this specifically about Youtube recommended videos in the sidebar?

I follow this account: @gerfingerpoken

Has not been censored at all on my newsfeed!

That is the same account covered by this article:

https://www.washingtonpost.com/business/economy/as-a-conserv...

Tweets on average more than 400x per day.

I'm pretty skeptical this is true. I don't think this has to do with Twitter censoring left or right wing accounts.

Always useful to see implementations of simple components, like epsilon greedy action choice:

    function nql:eGreedy(state, testing_ep)
        self.ep = testing_ep or (self.ep_end +
                    math.max(0, (self.ep_start - self.ep_end) * (self.ep_endt -
                    math.max(0, self.numSteps - self.learn_start))/self.ep_endt))
        -- Epsilon greedy
        if torch.uniform() < self.ep then
            return torch.random(1, self.n_actions)
        else
            return self:greedy(state)
        end
    end

There's definitely some examples of useful ops here:

    # 1. Define our computation as some op
    def useful_op(input_a, input_b,
                  use_clipping=True, remove_nans=False, solve_agi='maybe'):
        # ...