taliesinb

When I was a teen I read Steven Levy's book "Artifical Life". It captured my imagination in a way that the biomorph program in a Blind Watchmaker also had. Ever since then I've played with toy alife models. Some of the attraction is just aesthetic - it's fun to set these things up and design the rules, kinda like a higher-order god-game. Another is the enticing fantasy that something like this will one day lead to AGI. A third is the occasionally fulfilled desire to see the fabled emergent behavior of evolutionary computations, where the agents learn to do something you never expected, or exploit some funky loophole to their advantage.

When I was an undergrad I created an alife system as an honors project. It was an ambitious project, with sexually or asexually reproducing agents controlled by recurrent neural nets. The agents had perception and the ability to move, reproduce, attack, co-operate, or interact with objects. It had an entity framework to make it possible to design things like programmable Skinner boxes, traps, feeders, it had feeding schedules, it had live visualization and a slick Qt interface to edit the entities inline or take control of specific agents and visualize their brains.

I was so proud of it.

My assigned supervisor, who was an austere Georgian category theorist, had zero interest. He compared it unfavorably to a virus. Anyway, it was so dispiriting an experience that I kind of swore off Alife after that. But the fun I had made me realize I didn't want to be a mathematician, so I suppose it played a useful role in my life after all.

Mathematica, which is a serious project, is largely written in Mathematica. Wolfram|Alpha, another large project, is built in Mathematica. Outside the company, https://emeraldcloudlab.com/ for example has built their platform on Mathematica.

> And by "serious", I guess I mean something with few enough constraints to have an identity of its own, rather than being "a thing you can do with Mathematica."

For research, a "thing you can do with Mathematica" is often what you want. But other than that you can put things you build in the cloud via APIFunctions (similar to AWS Lambda functions), or call out to them via wolframscript, or talk to kernels directly via MathLink or LibraryLink, or over sockets via ChannelListen.

> Another limitation is the data input. Someone needs to rethink it seriously.

That's vague, but I imagine you mean importers. Certain built-in importers aren't as good as they should be, like CSV and XLS are memory hogs and die on relatively small amounts of data (at least they used to, I haven't checked recently). The HDF5 importer is now pretty good, and for large datasets it's a good choice for scientific computing anyway.

> Programming languages/platforms are network goods. IMHO, Mathematica has tried to swim against this fact and has failed.

No arguments there. I think it would be great if we could open source at least parts of it, because no doubt new life would be breathed into cobwebby parts of the codebase and various pet peeves fixed. But Mathematica still dominates the computer algebra space despite being closed source, and probably will continue to do so for a while.

See http://reference.wolfram.com/language/guide/NeuralNetworks.h..., also look at Examples > Applications under http://reference.wolfram.com/language/ref/NetTrain.html for some worked examples. Fun example of live visualization during training (very easy to do, will get even easier in future versions): https://twitter.com/taliesinb/status/839013689613254656

Uh, nope, that was speedup on 64 GPUs (or CPU cores, can't remember). i.e. it scales linearly, something that TF hasn't always been good at v other frameworks. I'm amazed a journalist with (I assume) basic technical competence could make this mistake.

Am I right in thinking that there is no bucketing going on here? In other words, each batch is fixed length, and a short-lived DAG is planned and then simulated with tf.while to accommodate the set of shapes in that particular batch? Are there any problems when the input shapes are wildly different in expense? For example, imagine a size-agnostic convnet. Maybe some of the images in the training set are small, others are large, how would that look in your framework, if it can be done? Is junk padding part of picture, to help match almost-equal tensors to allow them to be batched?

Sorry, I have to ask.. any relation to Dan Fortunato? Maybe that's just a really common name in Boston.

Visualizations can help, but more in the sense of rapidly iterating through bespoke visualizations to generate and test hypotheses. And to do that efficiently you don't need a slick tool that does one kind of visualization, you need a slick 'grammar' to efficiently build new visualizations with, Bret Victor style.

You're actually objectively† wrong, at least about why it is broken storytelling (which I don't think it is).

For example, you're complaining about: the fact that there was a protagonist, the plausibility of the sci-fi premise, some minor self-plagiarism, a political slant you don't agree with, a misremembered plot point about why there were several ships, a debatable concern about medical protocol in an unprecedented situation, and other sundry things.

These things are all essentially irrelevant to well-crafted storytelling.

If the movie didn't work for you, the things you were noticing are more like symptoms of the problem. Which by the sound of it was that you were just bored by the aesthetic choices the movie made.

I mean, I was totally transfixed and fascinated for almost the entire duration of the film. It might be that the contemplative mood, haunting music, deliberate editing, subtle acting, subdued lighting, etc. all provoked a kind of melancholic trance in me that made me insensitive to any story-telling deficiencies, it's hard to tell, but also, I don't care and the point is you're wrong.

† I justify the word objectively by claiming that the ensemble of extant human brains responds pretty predictably to these cultural artifacts we call stories. And there are people whose profession it is to understand how stories work and how to tell them. And I've grown up around several such people, and paid attention to what they have to say. And yeah I have the gall to think your post is just unbearably wrong.

Makes you wonder how many very talented people are stuck doing shitty things in any industry because things just didn't line up for them. At least they have for Heisserer, now. He's like the inverse of Lindelof, who turned out after all the hype to be a phony.

Short article for Guardian post Trump victory: https://www.theguardian.com/commentisfree/2016/nov/09/donald...

Longer article in Harpers about establishment response to Sanders, not directly relevant but uses the same class-based analysis: http://harpers.org/archive/2016/11/swat-team-2/

Talk about his most recent book "Listen, Liberal": https://www.youtube.com/watch?v=ibDX92b5cnY

> We like to emphasize that we have not derived the theory of modified Newtonian dynamics as proposed by Milgrom. In our description there is no modification of the law of inertia, nor is our result (7.43) to be interpreted as a modified gravitational field equation. It is derived from an estimate of an effect induced by the displacement of the free energy of the underlying microscopic state of de Sitter space due to matter. This elastic response is then reformulated as an estimate of the gravitational self-energy due to the apparent dark matter in the form of the integral relation (7.40). Hence, although we derived the same relation as modified Newtonian dynamics, the physics is very different. For this reason we referred to the relation (7.43) as a fitting formula, since it is important to make a clear separation between an empirical relation and a proposed law of nature. There is little dispute about the observed scaling relation (7.43), but the disagreement in the scientific community has mainly been about whether it represents a new law of physics. In our description it does not.

http://link.springer.com/article/10.1134/S1063776106010092

http://journals.aps.org/pra/abstract/10.1103/PhysRevA.85.012...

I just watched https://www.youtube.com/watch?v=ku6jVUJONZAm which is a recent lecture in which Matthew Fisher goes into some detail for a predominantly physics theorist audience. He doesn't actually finish his whole talk, but what he does cover is still pretty interesting to me as someone just trying to follow along with my rudimentary recollections of college quantum mechanics.

So {ortho,para}-pyrophospate is a big part of his grand hypothesis. This is analogous to {ortho,para}-water, something I hadn't heard about but which is apparently pretty famous, except with phosphorous instead of hydrogen. And as part of discussing this he mentions a study, presumably http://www.biophys.ru/archive/h2o-00011.pdf, that claims that passing water vapour through charcoal can change the ortho/para ratio from the normal 3:1 to 12:1 and then down to 2:1 for hours. That's just amazing to me, that liquid water can effectively remember a prior interaction for hours. Like that Japanese crank scientist's claims about water memorizing poetry or something, but, you know, actually real.

Fischer's proposed explanation is that ortho-water (and orth-pyrophospate) can't "sit still" because of the way that the aligned nuclear spins of the hydrogen (phosphorus) atoms cause the angular momentum to be quantized in a way that locks the angular velocity to be non-zero. And that discourages the ortho-molecule from being adsorbed or participating in catalysis -- measurable things which can clearly have biological effects. And from the nuclear spin, which is otherwise very long lived!

That's just crazy! Metastable water lasting for hours? I just smiled.

The other stuff may even be a red herring, but even this basic idea is so exciting if true. Here's a suggestive quote from the Russian article:

> Ortho and para water are expected to have notably different physical-chemical properties and, like ortho and para hydrogen, have different magnetic properties. The absence of the magnetic moment in para modification implies its complete insensitivity to the magnetic field, which could be of use in magnetic resonance imaging. The OP separation procedure realized is quite straightforward and may occur in nature—in soil, atmosphere, living organisms, and cosmic objects. The scope and the role of this phenomenon are yet to be studied.

Saying "a neuron per pixel" doesn't mean anything, really, that way of thinking isn't helpful unless you're looking at small multi-layer perceptrons. The right way to think about things is that you have tensors and layers that compute new tensors from old tensors.

A 1x1 convolution only 'sees' the feature channels of a pixel, and does the same thing to each pixel. So a 1x1 convolution on a grayscale input (e.g. a 1x28x28 tensor in the case of MNIST) does nothing, basically, other than scale and bias every pixel by the same linear function. It doesn't "force the network to learn" anything, it's just totally pointless.

One of the uses of 1x1 convolutions is to collapse the feature dimension when you're deeper in the network (e.g. 100 channels to 10 channels) to reduce number of parameters subsequent layers need operate on. It's a "channelwise fully connected layer".

I think you're thinking of (and perhaps what the author was thinking of) is the practice prior to convnets of collapsing the image into a vector and then doing a fully connected layer on it. That indeed doesn't exploit translation invariance of natural images, requires the net to learn the same features in every required spatial position at great expense, and so on. But that has nothing to do with 1x1 convolutions.

Why would it be against the original principles of LeNet?