Hacker News Reader

imurrayon Project Gutenberg blocks access from Germany

"Verlag" is just the German word for publishing house.

imurrayon Speech and Language Processing, 3rd ed. draft (2017)

Why link to the pdf?

The webpage https://web.stanford.edu/~jurafsky/slp3/ links directly to the PDF, and gives context and other download options. It's not so easy to go back from the PDF to the webpage.

Mods: I'd change the link, and the title to "...3rd Edition draft".

Everyone else: please stop linking to PDFs when there is an obvious html page to link to instead.

imurrayon Things I Wish I'd Known About Bash

You might want to search with

    /   -p

imurrayon Attention Is All You Need

I suggest changing the link from the .pdf to the web page: https://papers.nips.cc/paper/7181-attention-is-all-you-need

It's one click to get the pdf from there. But you also get a plain webpage with abstract, citation details, and so on, which you can't get back to from the PDF.

In general it's good to knock the ".pdf" off the end of all papers.nips.cc links. Similarly turn /pdf/ links on arXiv into /abs/ links, and replace "pdf?" in openreview.net links with "forum?".

imurrayon Switching to the Mutt Email Client

if you do inline quoting, MS Outhouse web client thinks that everything after the beginning of the quotation is a copy of the previous message and folds the whole thing, meaning your recipients get back what looks like an empty message. Really.

Gah; that explains it. Occasionally I get a reply saying "you just sent me a blank email", while the text of my email is quoted right there in the email they've just sent me!

imurrayon 3D printing for mathematical visualisation [pdf]

Coincidentally I just went to a nice exhibit in Edinburgh co-authored by Henry Segerman: http://www.maths.ed.ac.uk/~aar/brilliantgeometry/ — the last day is tomorrow for Edinburghers.

He has a book with supporting material here: http://www.3dprintmath.com/

imurrayon A Conceptual Introduction to Hamiltonian Monte Carlo

Implementations of HMC are approximating ideal Hamiltonian dynamics by discretizing time. The "symplectic" integrators that are used have the property that they correspond to simulating continuous-time Hamiltonian dynamics exactly on some other energy surface, which is not too far from the one you had in mind. So, in some sense, an approximation like the grandparent had in mind is the usual case.

You are right that a bad approximation of the surface could lead to proposals that get rejected a lot. But at least the dynamics are likely to be stable. It could be worth trying.

imurrayon A Conceptual Introduction to Hamiltonian Monte Carlo

Yes, you need the gradient. And despite what the autodiff people say, that might be hard and/or not very useful in practice, if your likelihood is based on running a lengthy astrophysical simulation.

You can run HMC on any differentiable surrogate surface surface. It's only the accept/reject step that has to use your real model and data. You might consider that if you have a less hairy approximation to your model.

Emcee is popular in astronomy, partly because of network effects, partly because it's a really nice package, and partly because the method's a great fit for some of the posteriors you have: often unimodal-ish even if the likelihood is nasty to compute, and often in not that many dimensions. It would fall flat on its face for something like a Bayesian neural network.

imurrayon Dual Numbers and Automatic Differentiation (2014)

I think you're assuming that "automatic differentiation" (AD) is the same thing as "using dual numbers", but AD is a family of methods including dual numbers. The blog post you link to says they're using https://github.com/denizyuret/AutoGrad.jl which uses reverse-mode differentiation not Dual numbers. That means they're performing the same operations (or nearly the same) as standard backpropagation.

Dual numbers are just the wrong approach for neural nets, which have many parameters. The amazing thing about backprop / reverse-mode differentiation is that you get the derivatives wrt all the weights in one reverse sweep through the network.

imurrayon Dual Numbers and Automatic Differentiation (2014)

> If I had to guess, I’d say that dual numbers may be slightly slower than backpropagation

The cost of Dual numbers (a form of forward-mode differentiation) scales linearly with the number of derivatives (just like finite differencing, but more accurate). Backpropagation, or reverse-mode differentiation, is a constant factor times the cost of a function evaluation. For neural nets with millions of parameters, backpropagation is going to be millions of times faster than dual numbers.

imurrayon Dual Numbers and Automatic Differentiation (2014)

If the function is analytic and supports complex numbers (usual in Matlab/Numpy/Fortran/...), but you're only interested in the real part, a quick hack is to abuse complex numbers and compute imag(f(x+i*epsilon))/epsilon. That one-liner can often give the right answer to machine precision for small epsilon. http://blogs.mathworks.com/cleve/2013/10/14/complex-step-dif...

For matrix-based code, a good note on derivative propagation is https://people.maths.ox.ac.uk/gilesm/files/NA-08-01.pdf — the symbols with dots on top are forward-propagated derivatives like dual numbers. The symbols with bars on top are back-propagated derivatives, which is what you want to compute lots of derivatives at once. "Machine learning" libraries like Theano, TensorFlow, and Autograd will compute the reverse mode operation for many linear algebra expressions automatically, and use reasonable libraries like BLAS+LAPACK or Eigen under the hood.

imurrayon Stop Using Arial and Helvetica (2013)

The article nicely demonstrates that .jpg is not a good format for screenshots of text.

imurrayon OpenNMT: Open-Source Neural Machine Translation

Neural translation has been really amazing and impressive. I stumbled into a way to be mean to Google translate this week: https://twitter.com/driainmurray/status/818934530207862784

OpenNMT can also run amok given totally unreasonable input too (example below). I'm being unfair, but the more serious point is that translation systems might go crazy in less obvious ways. While fluency is greatly improved, they certainly don't always get the meaning correct. I'm excited to see what people will do to better diagnose when we can trust the translation.

Input: "Toll ach toll toll ach toll toll toll toll toll toll toll toll toll toll toll toll toll toll toll toll toll toll toll toll Deutsch."

Output: "Tell me great great, great great, great great great great great great great great great love great ? ? ? great love ? ? ? ? ? ? ? great ? ? ? ? great love ? ? ? great great ? ? ? great great ? ? ? great great ? ? ? great great ? ? great great ? ? ? great great ? ? great great ? ? great love ? ? Good ? ? ? great love ? ? great love ? ? Good love ? ? great love ? ? great love ? ? great love ? ? great love ? ? great love ? ? great love ? ? great love ? ? great love ? ? great love ? ? great love ? ? great love ? ? great love ? ? great love ? ? great love ? ? great love ? ? great love ? ? great love ? ? great love ? ? great love ? ? great love ? ? great love ? ? great love ? ? great love ? ? great love ? ? great love ? ? great love ? ? great love ? ? great love ? ? great love ? ? great love ? ? great love ? ? great love ? ? great love ? ? great love ? ? great love ? ? great love ? ? great love ? ? great love ? ? Good ?"

[Each ? was a musical note glyph, but HN ate the unicode character.]

imurrayon Apple loses patent retrial to the litigious VirnetX, ordered to pay $300M

https://news.ycombinator.com/item?id=11862476

imurrayon Gorgonia: a library like Theano or TensorFlow, mainly written in Go

You could possibly try https://twitter.com/AlisonBLowndes

imurrayon WaveNet: A Generative Model for Raw Audio

Would delete this post if I could. Was a request to fix a broken link. Now fixed.

imurrayon Stacked Approximated Regression Machine: A Simple Deep Learning Approach

I'm going to check back later to see if anyone manages to reproduce it. Perhaps by the time it's presented at NIPS.

A twitter conversation reflecting some scepticism, but agreeing it would be interesting if it all checks out: https://twitter.com/fchollet/status/771862837819867136

imurrayon This JPEG is also a webpage

Right-clicking on the image and selecting "View Image" (Firefox), or "Open image in a new tab" (Chromium), gives the webpage, not the image. I can see why that happens: the menu items just open a URL and don't force it to be an image. However, it was a bit disorienting.

imurrayon Linear regression

I agree: If you have data X and outputs y and must have a linear function of X to minimize square error on y and only y, then yes we have defined our task.

I think it's a lucky engineer that has their end-goal defined as such a crisp mathematical task. Defining what is an optimal way to do a fit, for the surrounding application, usually requires thought. Often the reasonableness of the curve does matter, for example if it will be evaluated in locations other than the training data. Also, there may be some choice in how to set up the problem. For example, maybe we're allowed to transform the inputs with some fixed non-linear functions to create extra features. Adding these extra features to the design matrix will improve fit, so an engineer might consider it, but they'll need to be careful about over-fitting if they care about a surrounding application.

imurrayon Linear regression

I'm not sure what you mean, but I think we're talking past each other.

Regression is a statistical term. If it's going to be used for an application involving data and real decisions, not understanding the statistical issues is irresponsible. That's not at odds with viewing it as a procedure to minimize square errors.

To clarify: There are probabilistic models (involving Gaussians and stuff) where some interpretations suggest least-squares estimation. That wasn't the distinction I was trying to make. Statisticians who hate probabilistic models will still say that it's important to think about what you put in X and y before you do X\y, and to be careful about what you can conclude from the result.

imurrayon Linear regression

If you're new to learning about regression and want to use it, I would prioritize the statistical issues. There are lots of computational details you could read about, like in this post. However in simple cases, fitting least-squares linear regression is one line of code:

    w_fit = X \ yy; % Matlab/Octave, X is NxD, yy is Nx1
    w_fit = np.linalg.lstsq(X, yy)[0] # NumPy equivalent

Knowing how to construct the "design matrix" X is the important bit. What are your features, how will they be pre-processed, and given what you've done, should you trust w_fit to be meaningful, or just useful for prediction?

If you've thrown in lots of features, hoping to get good predictions, you may want to do "regularization". The computation is no harder for simple versions. One way to do "L2 regularization" is to add some extra rows to X and yy before fitting the weights:

    % Matlab/Octave... the NumPy is nearly as simple.
    [N, D] = size(X);
    lambda = 1; % regularization constant
    X = [X; eye(D)*sqrt(lambda)];
    y = [yy; zeros(D,1)]

Or you could use an actual stats package like R's libraries that will contain a lot more care. Again, you need to understand the statistics, for example how you should select lambda. Then you'll want to know how to diagnose your model and whether you should trust it.

It's fun to work out the least squares maths and implement these things from scratch. But it has been done, and it's not what matters for most applications.

EDIT: a discussion with good links on linear models and beyond from the other day https://news.ycombinator.com/item?id=12237998

imurrayon Metropolis Algorithm and Bayesian posterior sampling

The thinning here is based on a misconception. If the purpose is plotting a histogram it doesn't matter if the samples are correlated. The bin heights are consistently estimated if you keep everything. Throwing away intermediate steps usually just introduces noise.

Thinning is a good idea if the samples are strongly correlated, and the computation to process them all would be better spent on running the chain for longer. Or if you don't know what computation you want in advance, and you can't afford to store everything.

(Also, as someone points out in the comments, the implementation of the Metropolis algorithm in this post is wrong.)

imurrayon MNIST Handwritten Digit Classifier – beginner neural network project

A lot of machine-learning papers are eight pages. Speech conference papers (heavy users of neural nets) are often only four. Some details aren't part of the main message, so don't make it in. Often code is available, and initialization and other tweaks can be found in there (even if you aren't going to use their code).

That said, there are also whole papers, even collected volumes, on initialization and other practical details.

Textbooks aren't always up-to-date with the latest practical knowledge, as deep-learning practice is moving quickly. Or they simply don't want to clutter their high-level maths descriptions with code-level implementation details. Teaching stuff is all about tradeoffs. I'm sure several books do mention the scale of weights for simple feed-forward weights though, as it's not an implementation-level detail, and it's probably been well known since the 1980s.

imurrayon MNIST Handwritten Digit Classifier – beginner neural network project

Nice. Recreating these methods in simple code for yourself is definitely the way to check you understand it. This demo looks nice, clean, and straightforward. (Although I'd rename or comment variables x and y, or give some sort of guidance on what way around the weight matrices are within the code itself.)

It's also worth checking out existing neural net code-bases to see what tricks they have. The fine details usually aren't in papers, and they're not all in the text-books either.

The first potential problem that jumped out at me in this code was the initialization:

    self.weights = [np.array([0])] + [np.random.randn(y, x)
            for y, x in zip(sizes[1:], sizes[:-1])]

If the number of units in a layer is H, the typical size of the input into the layer above will be √(H). For large H, the sigmoid will usually saturate, and the gradients will underflow to zero, making it impossible to learn anything. There are some tricks to avoid the numerical problems, but even if you avoid numerical-underflow, things probably aren't going to work well.

I'd multiply those initial weights by a small constant divided by the square-root of weights going into the same neuron. For multiple layers you might consider layer-by-layer pre-training. For other architectures, like recurrent nets, definitely find a reference on how to do the initialization.

PS I would definitely add a test routine to check that the gradients from back-propagation agree with a finite difference approximation. It's so easy to get gradient code wrong, and it's so easy to test.

imurrayon GIMP 2.9.4 and our vision for the future

No it's not the same amount of clicking. From Gimp 2.7 to 2.8 something that used to be ctrl-s and close window went to ctrl-shift-e, and answering four dialog boxes before getting the image saved and the window closed. Around the time Gimp 2.8 came out, I wanted to edit and save around 50 images (in away that wasn't easy to automate). That (without the plugin that gives the old behavior) would be 200 dialog boxes to deal with that I didn't have to before.

No it wasn't literally painful. But frustrating. Nearly as frustrating as people not accepting that people legitimately use software in different ways.

imurrayon GIMP 2.9.4 and our vision for the future

I totally see that it makes sense for some work flows, including the ones Gimp is probably most meant for. It was frustrating that for the people whose workflows were messed up, the developers were totally opposed to doing anything about it though. Kudos to the plugin author for rolling up her sleeves and providing a way out. Also kudos to the gimp developers for providing a plugin architecture that made such a customization an option.

imurrayon GIMP 2.9.4 and our vision for the future

After ctrl-shift-e I have to hit enter in response to different pop-up dialogs three times, and then still answer a fourth confirmation when I quit gimp. If I'm editing a bunch of files that's not fun. With the Saver plugin I have it set up so I just hit ctrl-s like I used to, and can exit without being bothered.

imurrayon GIMP 2.9.4 and our vision for the future

Gimp 2.8 frustrated me: I could no longer open a .png, edit it and easily resave it back to .png. The Gimp developers knew better and made the UI strongly favor saving as .xcf, which makes sense in some use-cases, but not mine.

I found this fix, which made quick uses of GIMP less painful for me: http://shallowsky.com/software/gimp-save/

I'm assuming the new save behavior persists in Gimp 2.9.x, but I don't know.

imurrayon Scipy Lecture Notes – Learn numerics, science, and data with Python

Don't forget the biggest edge Python has over MATLAB for numeric simulation, i.e., you can prototype your work as quickly, and for the same program, convert the slow portions into C or Fortran without the need to rewrite the whole thing.

That statement applies equally to Matlab/Octave. You can link C/C++/Fortran routines into Matlab code using its "mex" interface. Octave also supports this interface.

Python is more flexible, but less simple to start out with because it has a more complicated ecosystem. For example, there are many different ways to link in fast code (ctypes/cffi/cython/numba/weave/f2py/...). Moreover, the data in a numpy array can be stored in memory in different ways (C or Fortran order, not necessarily contiguous, ...), which requires some care when passing it to external routines.

imurrayon Introduction to Scientific Python

Autograders are a mixed blessing. At my University some of our courses have autograders, and I think they can be great. There is a compiler course where the main goal is to create a compiler that has a specific set of measurable features. Encoding those features in a test-rig and making it available seems appropriate.

I deliberately don't use Autograders for my assignments though. I want students to be able to figure out for themselves whether to be satisfied with an answer, and work out how to test it. And I suspect many employers want the same. If a student asks me "is that right", I try to remember not to tell them, but to ask questions that will help them figure it out for themselves. Similarly, providing a test-rig that answers whether a piece of code is right is sometimes inappropriate.