Hacker News Reader

ajtullochon One parameter is always enough [pdf]

But since everyone (implicitly or explicitly) specifies the precision of the coefficients (eg fp32, fp64, int8, etc), it’s a complete straw man you’re arguing against.

ajtullochon GDPR for lazy people: Block all European users with Cloudflare Workers

https://en.m.wikipedia.org/wiki/Capital_strike

ajtullochon Executing gradient descent on the earth

One big problem with the conclusion is that intuitions from low dimensional spaces often don’t carry over to high dimensional spaces. e.g. common example is how the volume of the unit hypersphere intersected with hypercube ratio goes to zero. One funny thing I saw once was something like “the real curse of dimensionality is how different optimization in high dimensional spaces is compared to low dimensional spaces”.

ajtullochon The Game of Everything, Part 1: Making Civilization

Related, and quite an interesting perspective: https://www.rockpapershotgun.com/2018/03/15/all-rise-and-no-...

ajtullochon Facebook open-sources Detectron

They exist and are (nearly all) open-sourced for CPU & Metal (I wrote some of them).

ajtullochon Facebook open-sources Detectron

Look at the first table in https://arxiv.org/pdf/1708.02002.pdf (the RetinaNet paper, code + models included in this release). There’s a substantial accuracy cost for YOLO vs other, similarly fast (within a factor of 2 or so), methods.

ajtullochon Facebook open-sources Detectron

FWIW we have some similar R-CNN models using this codebase running in the FB apps on iOS/Android with Caffe2 - with some neat bells and whistles (eg full execution of the compute graph on the device GPU where available via Metal/OpenGL). I’ll look into adding a mobile tutorial for Detectron if you’re interested.

ajtullochon AWS DeepLens – Deep learning enabled video camera for developers

It's not very much. 100 GFLOPs is about 1-2 Haswell/Broadwell cores (32 FLOP/s * ~2 GHz AVX frequency). Even 2015 era mobile GPUs like the Apple A9 GPU (iPhone 6S) is higher throughput (~150-200GFLOP/s achievable).

If you're executing these models at 30FPS on an HD stream (say ~1mm pixels/frame), you have roughly a 100 * 10^9 / (30 * 10^6 ) = 3,000 flops/pixel budget if you're operating at peak, which is about the same flops/pixel of a 16 channel -> 16 channel 3x3 conv. In practice of course you can usually operate on a much smaller spatial input.

ajtullochon Cavium Is an Arm Server Contender

They don't for fp32 or int8 workloads until SVE IMO. High end Skylake as an example has 2 AVX-512-F ports (and not considering AVX-512-4FMAPS), which give you do 64 FLOP/cycle (512 / 32 * 2 * 2). Even 2 NEON units, assuming FMA's execute in 1 cycle/unit, gives only 16 FLOP/cycle. For ML inference workloads you can even get away with lower precision and even do away with floating point (which helps motivate AVX-512-4VNNIW).

ajtullochon Singular Value Decomposition for Programmers

Not with randomized methods (which you should almost always use IMO).

ajtullochon Why Is NumPy Only Now Getting Funded?

A few of the Eigen core devs (Benoit Jacob, Benoit Steiner, ...) work at Google.

ajtullochon Show HN: Bender – a Deep Learning framework for iOS

- https://github.com/xmartlabs/Bender/blob/master/Sources/Meta...

- https://github.com/caffe2/caffe2/blob/64e52e474/caffe2/contr...

ajtullochon Implementing a Principal Component Analysis In Python (2014)

If anyone is interested, a much faster technique is using the randomized methods from https://arxiv.org/abs/0909.4061 - see e.g. http://scikit-learn.org/stable/modules/generated/sklearn.dec...

ajtullochon Re: What is acceptable for -ffast-math? (2001)

-ffast-math (well, really -funsafe-math-optimizations) is also essential for FP vectorization on ARM CPUs. Compare the generated code of a real-world ARM function with/without funsafe-math-optimizations - https://godbolt.org/g/P4C9qm (after, ~70 instructions) vs https://godbolt.org/g/blAe2q (before, ~270 instructions). As GCC's documentation states (https://gcc.gnu.org/onlinedocs/gcc/ARM-Options.html):

  If the selected floating-point hardware includes the NEON
  extension (e.g. -mfpu=‘neon’), note that floating-point
  operations are not generated by GCC's auto-vectorization
  pass unless -funsafe-math-optimizations is also specified.
  This is because NEON hardware does not fully implement the 
  IEEE 754 standard for floating-point arithmetic (in 
  particular denormal values are treated as zero), so the use
  of NEON instructions may lead to a loss of precision.

ajtullochon Intel will add deep-learning instructions to its processors

https://github.com/Maratyszcza/NNPACK/blob/master/src/x86_64...

(from https://github.com/Maratyszcza/NNPACK) is also a pretty reasonable AVX2 GEMM implementation written in a Python assembler, it's a bit easier to follow than the OpenBLAS kernels and has very reasonable performance vs OpenBLAS/MKL.

I second the guidance to just use a BLAS library. I'd guess you'd see a 2-10x speedup at least. It looks like you're only implementing a classical MLP library (which is fine for learning, but not the best choice for something like this, integrating VW into Redis would be a much better choice IMO).

ajtullochon Google Sprints Ahead in AI Building Blocks, Leaving Rivals Wary

"It wraps cuDNN, which is faster than Tensorflow."

TF uses CuDNN, so this is a nonsensical statement.

ajtullochon NNPACK – acceleration package for neural networks on multi-core CPUs

This is a really great piece of software, awesome stuff Marat! For those folks running convolutional neural networks on CPUs in production, it is a substantial enabler.

ajtullochon Intel Xeon D 12 and 16 core parts launched: first benchmarks

PeachPy is great, thank you @Marat for this project.

ajtullochon Introducing OpenAI

Congrats! It's a brilliant team, looking forward to great things.

ajtullochon FunctionalPlus – helps you write concise and readable C++ code

I would recommend instead Facebook's folly::gen library, which provides these functional combinators in a high-performance fashion, by expressing them as streaming combinators. You get all the map/filter/take/drop/reduce primitives, but in a much cleaner way.

See https://github.com/facebook/folly/blob/master/folly/gen/test... for an example.

ajtullochon The Unreasonable Effectiveness of Random Forests

That's actually not the case - that's one effect, sure, but when you compared the compiled trees to the flattened trees (the strategy just described), the compiled trees are substantially faster across a range of parameters. See http://tullo.ch/articles/decision-tree-evaluation/ for a detailed evaluation.

ajtullochon The Unreasonable Effectiveness of Random Forests

The article mentions that "The main drawback of Random Forests is the model size. You could easily end up with a forest that takes hundreds of megabytes of memory and is slow to evaluate."

A cool trick for speeding up trained random forests/gradient boosted decision trees is to dump the tree to C/ASM, compile it, and dlopen it as a function pointer. Depending on the model and the architecture, you can get an 8x speedup relative to an optimized C implementation that walks the tree.

There's an implementation for scikit-learn with benchmarks at https://github.com/ajtulloch/sklearn-compiledtrees if you're interested in this technique.

ajtullochon Neural Networks, Types, and Functional Programming

https://github.com/ajtulloch/dnngraph is an application of these ideas - it's a Haskell DSL for building DNNs for later execution by Caffe/Torch.

ajtullochon Effectful Haskell: IO, Monads, Functors

The Haskell monad typeclass _is_ (modulo technical details) a category-theoretic monad on the category `Hask`.

As Edward Kmett says in more detail in [1] - the point of calling a monad a monad is not to confuse the reader, but to unlock 70 years of documentation on the concept for the reader. How many programming concepts/tools/libraries do you use that have 70 years of documentation?

Being abstract is profoundly different from being vague.

[1]: https://yow.eventer.com/yow-2014-1222/stop-treading-water-le... (around the 20 minute mark).

ajtullochon The Unix Philosophy

This is a false choice - you can serialize/deserialize Thrift objects to/from JSON. I think the choice here is between having a clearly-defined /schema/ or not.

ajtullochon Abstractions my Deep Learning word2vec model made

Nit: gradient boosting isn't an 'augmentation' of random forests - if anything, it's the other way round. AdaBoost is from 1995, the GBM paper was 1999, and Breiman's random forest paper in 2001 explicitly couches it as an enhancement to AdaBoost.

ajtullochon How to implement a neural network: Part 1

The algorithms haven't (fundamentally) changed.

ajtullochon Show HN: Hydrogen Brings Light Table to Atom

Awesome - and very cool to see ITorch getting some love. Thanks very much @willwhitney.

ajtullochon Why GEMM is at the heart of deep learning

FWIW, FFT-based convolutional methods aren't better for a lot of common sizes (including a bunch of AlexNet/VGG/GoogLeNet sizes). http://arxiv.org/abs/1412.7580 has a pretty thorough evaluation of im2col + gemm vs fused gemm (cudnn) vs fft.