spi

Yes that might be the case. In my case I mostly trained big (tens to hundreds of millions of parameters) networks mostly made of 3x3 convolutions, and I think the V100 has dedicated hardware for that. Then as I mentioned you can get a further 2x speedup by using half precision.

If you train smaller models, or RNN, you probably lose most of the gains of dedicated hardware. But I guess that for this same reason the experiments in the article are little more than a provocation, I don't know if you could train a big network in finite time on M1 chips...

That said, of course, if the budget was mine, I wouldn't buy a V100 :-)

"Similar performance" still means 30%-50% slower [1] and half the RAM, not really that comparable.

For much closer performance you should get a 2080ti, which should be roughly comparable in speed and have 11GB [edit: wrongly wrote 14GB before] of memory (against the 16GB for the V100). Price-wise you still save a lot of money, after quickly googling around, roughly $1200 vs. $15k-$20k.

But you still lose something, e.g. if you use half precision on V100 you get virtually double speed, if you do on a 1080 / 2080 you get... nothing because it's not supported.

(and more importantly for companies, you can actually use only V100-style stuff on servers [edit: as you mentioned already, although I'm not 100% sure it's just drivers that are the issue?])

[1] I've not used 1080 myself, but I've used 1080ti and V100 extensively, and the latter is about 30% faster. Hence my estimate for comparison with 1080

I fully agree with this. As a data scientist, I always think that this is a "natural" consequence of one of the main (if not _the_ main) metric used to evaluate machine translation algorithms, which is BLEU: https://en.wikipedia.org/wiki/BLEU

According to this metric, if you have a moderately long sentence like "I am not the person who said the president should be reelected" and your translation missed the "not", you would still get a score of 11/12 ~ 92%. And, as far as I know, word order doesn't even matter, so "I am the person who said the president should not be reelected", while wrong, would get a perfect score.

Of course these are rather artificial examples, and in general machine translation algorithms and their evaluation work because it's "easier" to create an algorithm that gets the right translation than one that, unintentionally, fools the metric systematically. Nevertheless if the research community used a metric that punished this kind of mistakes more strongly, I suspect that over time a few new algorithms could come up that improve on this specific point.

Alas, I don't know of any such metric (nor I would know how to design one, of course, otherwise I'd publish it ;-) ).

Perhaps more importantly than the limit on 200/2000 samples, the main difference from your code is that in the paper they only use the 10 first PCA (i.e. X is 10-dimensional instead of 784-dimensional). The fact that they do tends to contradict their claim that the proposed algorithm scales "weakly" with dimension, but for using just 10 dimensions (chosen linearly from the input ones) 76% is not that bad.

Personally, my doubts are more in line with the other ones that were raised here, namely tdj's comment on not building on existing literature (the feeling of "this just looks too much like a classification tree" is very strong while reading), and scalability issues. I also tried something vaguely similar in the past (but using iterated linear discriminant analysis instead of hypercubes / classification trees, and convolutional networks instead of fully connected), but never even finished the prototype because it was so terribly slow even on MNIST that nobody in their sane mind would have used it (I could only load into RAM a few samples at a time, which made it messy).

In any case, it's a pity they didn't try to use these weights as initialization for the "usual" backpropagation, it might have lead to interesting results (especially if in doing so they extended to the whole of the input and not just the first 10 PCA - or at least on ore than 10).

Same disclaimer as v64: my background is also mathematics, but not in this field.

As he says, what was proved so far is that: a_0 < a_1 <= p <= t <= 2^(a_0) The first inequality is by definition (a_1 is "the smallest cardinal greater than a_0), the second one is a theorem by Hausdorff, the third one is supposedly easy, the last one is trivial. Also, if the first one is an equality, then the second one is, too.

The continuum hypothesis claims that they are all equal; that's been proved by Cohen to be independent of what you call "A" (and is generally called ZFC), by "Forcing" which is not easy, technical, and has won Cohen a Fields medal... And you're right in saying that what is proved in an axiom set is true in any larger axiom set.

However, the main point is that "we cannot have p<t, and hence p=t" is false; that is the whole point of indecidability. A statement can be "true", "false" or "not decidable". Many people apparently expected the p=t question to fall into this last group (with some enlarged axiom set "A+" permitting to prove the strict inequality).

And no, the statement "The continuum hypothesis is undecidable in ZFC" is proved within ZFC, otherwise it wouldn't have been proved at all... To give a trivial counter-example, as a corollary of Cohen's proof, the axiom choice "ZFC+CH" (ZFC + continuum hypothesis) is as valid an axiom set as ZFC is. Within it, you can certainly prove that "ZFC implies CH" since... CH is simply true.

Very nice article! I've been willing to delve into Bézier curves for a while now, and this looks like an awesome reference to start.

I'd just like to make an annoying mathematician comment on the beginning (paragraph 3): you might want to point out that the first definition you give of Bézier(n,t) is, for any n and t, nothing else than the constant function 1 (that, by the way, is an absolutely fine polynomial, unlike the statement above that if f(x) has no x then it's not a polynomial).

You can see this by noting that it's nothing else than $((1-t)\cdot t)^n = 1^n = 1$. If you insist on being fancy, you can say that the i-th term in the sum is the probability of a discrete random variable X with binomial law being equal to i, so the sum over all i must be 1 (here the binomial law has parameters (n,t)).

By the way, this last thing has the nice interpretation that each point of the Bézier curve is just the expected value of $p(X)$, where $p(0), \dots, p(n)$ are the points you are interpolating - by changing t you are just changing the parameter of the random variable.