taliesinb

The most interesting alternative to the Planet Nine hypothesis (and to my mind a more attractive one) is that there is a self-organizing dynamical process that applies to large ensembles of bodies in highly eccentric orbits that causes their argument of perihelion to become correlated.

There is a great SETI talk by the astronomer Ann-Marie Madigan that explains the idea, it should be easy to follow if you've ever taken a mechanics course: https://www.youtube.com/watch?v=YW0Zb5gY0HA

Hehe, yeah. Worse, it's a wrong theory. But still kinda interesting, no harm learning about it.

Agree a modern SimEarth would be awesome.

Cool hack. So you had to do multiple laps? It's interesting that dead reckoning actually worked, we had lacquered wooden floor in my house and the slip was quite bad, dead reckoning never worked.

> We've seen that kids are pretty comfortable with the idiom of "Pages."

Hey man! Yes, the pages are a really good idea. Among other things, it seems like a great way of sowing the seeds of structured programming.

Can't wait for the game to be released!

That's too easy. Think of a game like Civilization. Granted, it's limited what you can learn from reading the in-game Civopedia, but probably thousands of kids have more awareness and interest in the historical progression of human civilization from playing that game.

And even if it is superficial awareness, those kinds of games exercise the fantasy-making faculties that make history itself interesting to learn about, even later as an adult.

Another example. There was this flop of a game in the early 90s, called SimEarth, that I have very fond memories of. It taught me about the Gaia hypothesis and the carbon cycle and stuff. Gameplay was pretty wacky though.

... segue to the wonderful LEGO Mindstorms, named after Papert's book, which had a graphical programming language very similar to Squeak, which itself goes back to Kay's dynabook, influenced by Papert.

OT, but I'm very grateful to Mindstorms, it took me to the next level when I was a teen, from QBasic to C. Not because of its visual block language, which I hated as I could already code, but because I found out about a thing called NQC (Not Quite C).

NQC was a hobbled dialect of C that you could use instead of the LEGO's visual language to program the RCX (the main computer brick with batteries in it). I'd never tried C before, and it was a revelation.

And it unlocked wondrous hacks, like using the IrDA transceiver on the RCX as a psuedo-lidar so that your robot could stop short of walls, it could even estimate distance. At the time, it was so cool to me. From playing with my stumbling robot trying not to hit obstacles I got interested in Rodney Brooks and subsumption architectures. So it just inflamed my passions in AI and programming at a critical part of my adolescence.

Ironic, though, that a Papert-inspired graphical language was the antagonist in this story. The breakthrough came from wanting to scale the walls of the little garden LEGO had built.

Yes, unfortunate that MXNet wasn't covered. It's in the happy Venn place of (fully cross-platform) ∩ (easy to embed) ∩ (flexible) ∩ (hackable).

* Cross platform: Windows, MacOS, Linux; CPU and CUDA. Though their CMake needs work.

* Easy to embed: straightforward C FFI, JSON for metadata and parameter serialization, no weird runtime.

* Flexible: not too specialized to vision. Static unrolling of RNNs possible now (with mirroring this can still be very memory efficient [0]), basic support for the fast new cuDNN 5 RNN layers [1] (contributed by colleague of mine). Dynamic unrolling is on the horizon I hear.

* Hackable: once you're familiar with the codebase, custom elementwise unary or binary ops = few minutes, custom layers = 1+ hours (depending on complexity). And if you can leverage mshadow primitives for your layer implementation, you don't even have to touch CUDA. Also fairly active on github, responsive to PRs etc.

[0] https://arxiv.org/pdf/1606.03401.pdf

[1] https://devblogs.nvidia.com/parallelforall/optimizing-recurr...

Yes, realized this and was editing my comment to reflect that but you beat me to it :). They did mention a previous signal from 2013, which got me confused with the spurious signal the SETI lecture mentioned.

Wow, amazing result. And talk about synchronicity - just last night I watched an interesting 2015 talk about the search for planets around Alpha Centauri using the radial velocity technique: https://www.youtube.com/watch?v=eieBXGpNYyE

The speaker even mentioned the previous incorrect HARPS announcement, which was later found to be an artefact due to the windowing function they used - a pretty embarrassing mistake. This new finding involves a completely different period: 11.2 days instead of the previous 3.24 day signal.

Also, link to the Nature paper for the lazy: http://www.eso.org/public/archives/releases/sciencepapers/es...

Just watched https://www.youtube.com/watch?v=ZV1-EN_MFtA, pretty amazing.

> I get the regularization part, but don't you get essentially the same regularization from using a sparse autoencoder? If the encoder realizes it doesn't have much information, it will turn on few units.

Putting a sparsity loss on z in a regular AE will encourage the code to have smaller magnitudes, and with relu those units will tend to saturate to zero, yes.

But the original point was that even a single continuous unit can be used to transmit an arbitrary amount of information. Not so much that this happens in practice, because the encoder and decoder would need access to something like modulo to do the most obvious kinds of cheating, but just that from an information theory point of view you can't really talk about how much information a continuous variable transmits unless you are transmitting it over a noisy channel and can measure entropies of distributions (and indeed you can formally derive how a given KL loss bounds the information transmitted by z).

> What I don't really intuit is: is it just basically doing regularization, or is the interpretation in terms of learning to infer the posterior meaningful?

Both, which I think is really nice. You can look at it either way.

The Bayesian interpretation is powerful because you now have a principled way to calculate p(x), which you didn't have before. And you can introduce multiple latent variables in your network (as long as no layers take inputs from both ordinary and sampling layers) and so you have some flexibility to do limited forms of graphical modelling that supports efficient forward inference and GPU acceleration. And the inference machinery can be trained via cheap backpropagation instead of expensive sampling.

VAEs can be extended to make the latent variables dependent. OpenAI's inverse autoregressive flow is one recent way that is particularly efficient: http://arxiv.org/pdf/1606.04934v1.pdf. Linear IAF is the simplest form of this, with it you can model normal z having an arbitrary covariance matrix.

But aside from that, there is an information-theoretic view on why you might prefer VAEs over AEs. In short, having p(z|x) not be point-mass (aka an ordinary AE) allows you to bound the information flow through the bottleneck. KL loss on p(z|x) forces the network to be honest about how much information it is cramming into z for the purposes of reconstruction.

To unpack that a bit: in theory, even a single real-valued latent variable z could store an arbitrary amount of information (if the encoder and decoder conspired cleverly enough). But if you make z stochastic, or in other words if your encoder's job is to calculate the parameters of a distribution from which you sample z, you're essentially introducing a noisy channel in the middle of your network, and you can then bound how much information is flowing across that channel. But to do that you still need to use KL divergence loss to encourage p(z|x) to approximate your chosen latent distribution, otherwise your encoder and decoder might cheat, e.g. by using near-point-mass z as a way to turn back into ordinary AEs again.

Or in deep learning speak, it's a form of regularization with a particularly rich and interpretable statistical motivation.

If you want to know more about the neural network stuff specifically there is a gallery of marketing examples at http://www.wolfram.com/language/11/neural-networks/?product=... and quite a few examples at http://reference.wolfram.com/language/ref/NetTrain.html (look under Applications near the bottom of the page).

The current selection of layers is biased a bit towards vision, but my colleague and I are working on recurrent networks as we speak, which will unlock networks that operate on text, audio, and other data of a sequential or temporal character. Hopefully that will land in a few months with 11.1.

> A question for Mathematica-folk: does Mathematica have support for direct manipulation like Toby Schachman's works?[] I find direct manipulation more pleasant than typing characters.

Locator comes to mind:

https://reference.wolfram.com/language/ref/Locator.html

https://youtu.be/6wOLBVFdek8?t=11s

And more generally Manipulate which makes it easy to whip together a let-me-explore-how-this-depends-on-that type of widget.

Yup, that is odd. IEA appears to be off the rails here. Making the adjective "conservative" a strange choice.

> Picking and choosing predictions in this manner to make it look like they have all been steadily increasing is misleading.

Those predictions are all sourced in the original article. As you say, though, pointing out a trend in the predictions from a set of scenarios doesn't on its own tell you anything.

> The last IPCC report I read predicted an increase of between 0.3C and 0.7C by 2035 with around 0.4C being the most likely. Nowhere near the the 3.5C by 2035. There is very little consensus for numbers that high.

Was that the fifth assessment report?

And where does the consensus 'currently' cluster for 2035? What climate models would be included? Under what emissions scenarios? I'm not too familiar with the state of the current literature.

The IEA may not sure the view of the author that "such an increase would remove habitat for human beings on this planet".

The current https://www.iea.org/publications/scenariosandprojections/ claims that "business as usual" would yield almost 4C by 2100 and 5.5C long term. Not sure how that squares with claim by author or which scenario they were referring to.

The big question I have with many of the scenarios (I think IPCC ones fall into this camp) is how much worse they will get when methane clathrates release is included. I'm not too familiar with the literature, maybe others know how well clathrate release is understood and what contribution it makes under various scenarios.

I feel the same way.

As history has shown, when presented with an existential threat to the group, humans are often willing to drop their immediate plans and sign up to fight. Perhaps we're wired that way.

Bring on UBI. And bring on a global 'war economy' to transition us to a zero carbon energy infrastructure.

Reminds me of this essay I read yesterday, which painted a pretty startling picture of how insane things could get:

http://www.flassbeck-economics.com/how-climate-change-is-rap...

Here's a snippet showing how rapidly predictions are escalating:

* Late 2007: The Intergovernmental Panel on Climate Change (IPCC) announces that the planet will see a one degree Celsius temperature increase due to climate change by 2100.

* Late 2008: The Hadley Centre for Meteorological Research predicts a 2C increase by 2100.

* Mid-2009: The U.N. Environment Programme predicts a 3.5C increase by 2100. Such an increase would remove habitat for human beings on this planet, as nearly all the plankton in the oceans would be destroyed, and associated temperature swings would kill off many land plants. Humans have never lived on a planet at 3.5C above baseline.

* October 2009: The Hadley Centre for Meteorological Research releases an updated prediction, suggesting a 4C temperature increase by 2060.

* November 2009: The Global Carbon Project, which monitors the global carbon cycle, and the Copenhagen Diagnosis, a climate science report, predict 6C and 7C temperature increases, respectively, by 2100.

* December 2010: The U.N. Environment Programme predicts up to a 5C increase by 2050.

* 2012: The conservative International Energy Agency’s World Energy Outlook report for that year states that we are on track to reach a 2C increase by 2017.

* November 2013: The International Energy Agency predicts a 3.5C increase by 2035.

Cool. I used to be big into artificial life. An old project of mine for anyone who is interested: https://github.com/taliesinb/floatworld/wiki

Nice. This is pretty similar to Mathematica's user interface functionality and Dynamic mechanism:

http://reference.wolfram.com/language/ref/Dynamic.html

http://reference.wolfram.com/language/ref/Manipulate.html

Michael Graziano's Attention Schemata Theory is in my opinion a pretty convincing way of thinking about consciousness. Here's a talk he gave about it:

https://www.youtube.com/watch?v=peHcu8LEgEE.

(warning: includes wacky ventriloquist act)

That's great to hear. Can't wait for TensorFlow to fully support Windows, that's the main thing stopping us from using TF instead of MXNet as a backend. Any news on that?

> I am still pretty upset and maybe even somewhat traumatized about all the previous times they've fucked me over in scenarios like this. It starts out great and then gets ruined.

I haven't done much development using Apple frameworks. I'm curious, where has this happened to you before?

That's the right paper. On page 8 you'll see the line "The last line is referring to all the changes is what we refer to as “Inception-v3” below".

You can stick to TensorFlow for training your networks, but if you want to deploy a trained network to iOS or macOS devices (and your network is expressible in terms of Apple's primitives), you'd be doing your users a disservice not to use the fastest and most energy efficient backend to do the actual inference.

I'll add that at this point, there's isn't much 'lock in' between different frameworks. Once you've trained, and if your primitives are available in the target framework, porting is just a matter of getting your weights and topology into the right format. Not too hard compared to the nitty gritty of gathering data, designing a network, and doing training and hyperparameter optimization.

It's worth noting than BNNS (the CPU library Apple released) defines just 3 layer types. It's clear they are for now targeting just convolutional networks (the kind you'd use for computer vision), and even then just the simplest possible kind (for example Alexnet, now ~4 years old).

There needs to be much more functionality to build a state-of-the-art vision network, such as Google's Inception v3, at least if you want to avoid writing your own layer definitions. Hopefully they'll expand this out later. In the foundation keynote they did mention using LSTMs for predictive text, so hopefully more advanced things are in the pipeline and just haven't been API-ified yet.

For reference, the deep learning library I'm working on will contain 17 layer types in its first version, with later versions probably asymptoting to double that. There are a LOT of things out there that deserve their own layer types.

EDIT: to be fair to Apple, since convolution, pooling, and fully connected layers are among the most costly layers, they are the most valuable to have efficient platform implementations for.

I don't know much about this sort of thing but I wonder if the ultimate performance would come with co-locating specialized compute with memory, so that the spatial layout of the computation on silicon ends up mirroring the abstract dataflow dag, with fairly low-bandwidth and energy efficient links between static register arrays that represent individual weight and grad tensors. Minimize the need for caches and power hungry high bandwidth lanes, ideally the only data moving around is your minibatch data going one way and your grads going the other way.

I wonder if they're doing that, and to what degree.

Seems very similar to Joyent's Manta.

The link I gave, and others, repeatedly didn't work when I tried them, but now they seem to work!