Hacker News Reader

Comment by argonaut | Hacker News Reader

As far as the recent deep learning boom is concerned, genetic programming is really out of favor. I don't really see it in any of the deep learning (or even machine learning, for that matter) literature/successes/research groups.

"Neural networks" are a really really overloaded term. A ton of stuff referred to as "neural networks" has little to do with the "neural networks" that are used in the machine learning community.

View on HN

You're spot on about genetic programming. I am a self-taught person who plays with anything that strikes my fancy; I learn by playing. I read all three volumes of the Artificial Life series from the Santa Fe Institute at the time (now there are more), and went in many directions in the 1990s - Fuzzy Logic, Expert Systems, ANNs, and Evolutionary Computation (GA (Genetic Algorithms) and GP Genetic Programming), and AL (Artificial Life) all fascinating. I found, and still find, genetic programming attractive even if it has not found its niche in the ML community. I think the CI (Computational Intelligence) community at large will eventually develop well-fitted uses for it. I was trying to use an FPGA and Koza's modified GP code to have the FPGA re-program itself as a GP evolved a better program than I originally wrote to kickstart it. I didn't get too far. This was 1996-97 though. Pretty much on my own then, not really much of an Internet to find information, especially esoteric information, or cheap many-gated FPGAs. Outside of ML, GP has found moderate success. One example is this paper (sorry behind paywall, so only the paper title here), that started with using expert data, tried ANNs, then ANNs and statistics, until it used a GP approach:

"A Computational Intelligence-Based Genetic Programming Approach for the Simulation of Soil Water Retention Curves"

I also use the term ANNs over just NNs to keep it to the silicon, and not wetware ;) Although, they did hook up a small ANN to a cockroach once, IIRC...

It has its niche applications. The only non machine vision application that comes to mind is one[1] that takes a pile of data, and evolves a model that fits it.

Generally were its actually being used they are a bit quiet on how they go about getting the results they do. While the genetic bit is easy, the secret sauce is in guiding learning/evolution that work for the particular problem domain.

[1]: http://www.nutonian.com/products/eureqa/

Yes, but all of the algorithmic advances in academia, and most of the advances at Google/Facebook, have been out in the open.

Yes, it is a shame people don't share their advances in science and technology for fear of losing market share usually. Sharing grows the market, and then there's more pie for everyone, and more work gets done to advance the field. Still that point, nor how successful GP is in the ML community, measures its current or future potential. The book I am working my way through now, in LFE (Lisp Flavored Erlang vs. Erlang, or Elixir), is "The Handbook of Neuroevolution Through Erlang" by Gene Sher [1]

Gene covers a lot of ground. Somebody has done some transliteration to Elixir too; I use LFE, since staying with Lisp bridges the gap between my GP work, and what Gene has done with Erlang and ANNs and EC. For GP, you really need to be able to create new forms with macros, or it is more in line with GP. To quote and excerpt from Robert Virding, co-designer of Erlang, and creator of LFE,addressing Elixir's macros or messing with Erlang's modules vs. LFE's or Lisp's macros on HN before:

  "There is syntactic support for making the function calls look less like function calls but the macros you define are basically function calls.

In Lisp you are free to create completely new syntactic forms. Whether this is a feature of the homoiconicity of Lisp or of Lisp itself is another question as the Lisp syntax is very simple and everything basically has the same structure anyway. Some people say Lisp has no syntax." [2]

  [1] http://www.erlang-factory.com/upload/presentations/536/ErlangConferencePresentation_2012.pdf

  [2] https://news.ycombinator.com/item?id=7623991

Just curious then, how are people optimizing network topology?

GSD, also known in the literature as "Graduate Student Descent."

I'm not even joking. Trial and error. Having good "intuition" about past ideas the basic building blocks to guide that trial and error. Reading research papers and seeing what other people did well with and using that.

As an aside, this is the principal reason I am skeptical of grandiose claims about deep learning.

Regularisation methods like dropout are often good enough that you can build a network with too many parameters (for the amount of data you have) and rely upon the regularisation to find the subset of that network that is actually useful. People have recently got good results from also randomly dropping weights, or even whole layers.

Probably also through some grid search. I've read (but not rememeber where) that Random Search gives very good results, even better than grid (in less time).

Reply on news.ycombinator.com