taliesinb

Funny story, I had almost exactly the same idea in 2013 and prototyped it in Mathematica (probably a lot of people have had this idea). Here's a screenshot of some random thread that I happened to visualize (comments were visible as tooltips in my rough and ready implementation): https://imgur.com/a/9xfw6cA (color represents common authorship, size is comment length). I didn't have the Javascript chops to do anything usable, let alone as polished as the linked Chrome plugin; kudos to the author!

Another thing I was doing at the time was visualizing Twitter and Facebook friend graphs, and the first insight there was to deliberately drop the 'ego' node, as it destroys any hope of effectively showing community structure of the friends themselves (it also carries no extra information as everyone is necessarily connected to the ego).

So perhaps a similar thing would be effective for threaded conversations: if you drop the root node, disconnected discussions will become separate trees in the visualization, which will help with efficient use of space, and the tail of singleton comments that no one replied to will become isolated disks that can space-fill the background. Switching to an actual rectangular tree layout as I used in my prototype will also help in that one consistent direction (down) represents nested depth.

Thanks! There’s an interview with Howard on the great Neuroscientists Talk Shop podcast: http://snrp.utsa.edu/Podcast/Entries/2016/9/20_Howard_Eichen...

Also a more recent discussion about how neural networks might memorize sequences: http://snrp.utsa.edu/Podcast/Entries/2017/9/21_Dean_Buonoman...

Whats a cheap and unobtrusive way to estimate the BSimple version of the noise scale in real time? Piggy back on ADAM's moving mean and variance estimates? Edit: I see that Appendix A has a method for the multi-device training setting, but I'm thinking of single device training.

This is an exciting work.

Gradient noise scale would be interesting to compute individually for each weight/bias. I'm curious how it scales as a function of layer depth, for example -- it could well be that different layers require very different treatment in terms of learning rate. Supporting this is that gradient magnitudes are typically very different for different layers, although they evolve similarly in time. Here's something I just whipped up in Mathematica:

https://imgur.com/a/HhCnlDL

Can you point at the autobatching packages? What strategy are they taking? Will they recognize opportunities to combine compatible operations within a given function body into a batch? Does one need a cost model for merging and splitting data into batches?

Also, what does an approach like bucketing even look like for the approach that Julia is taking? The idea there of course is to have 'slop': to combine many similar examples whose tensor sizes differ by small amounts, and to carefully define all your primitive operations such that they can can ignore the padding used to combine similar tensors into a uniform shape. Doing this requires awareness of the tensor sizes all the way back to the way you sample the training data, so I don't see how compiler magic can achieve the same performance as you get from bucketing.

Of course, bucketing becomes more complex for things like trees and graphs and other higher-level objects. And bucketing, theoretically, can bias into your gradients, if there is any correlation between the gradient of an example and its tensor shape.

Thanks. I'm more familiar with this being called model parallelism (exploiting parallelism in M) vs data parallelism (exploiting parallelism in v).

Can you elaborate on what you mean by "parallelizing those against cores" vs "parallelizing across samples in a batch"?

State reset is the kind of unique thing computers can do that we should be exploring and developing, rather than shying away from.

This is very cool work. Well done to team!

“Mapping” the state space has seemed like a really exciting direction to me and I’ve been puzzled to see so little work in this vein (I’d be interested if anyone knows of other examples).

Just to add, Kenneth if you are reading: we met at GECCO 2017, hi again, and really nice work! (I remember first learning about map-elites there, a simple yet beautiful idea from another of the co-authors. Satisfying to see it play a role in RL!)

Did you take a look at http://www.wolfram.com/language/fast-introduction-for-progra...?

Anyone here have industry applications of graph networks that they can talk about?

The Chemical Safety Board produces some pretty fascinating videos that reconstruct the chains of events that led to large industrial accidents in chemical plants. The videos normally conclude with recommendations for voluntary actions industry can take to prevent similar incidents in future. They basically summarize the results of detailed investigations by CSB agents.

The videos are well produced and, apart from those accidents that involve injury or death, they are quite entertaining if you happen to be in a forensic frame of mind.

Here’s an example of one of their videos: https://youtu.be/_icf-5uoZbc

The passage about implanting human organoids into primates reminded me of this YA scifi novel called Eva that I read as a kid. Girl in a car accident gets her brain transplanted into a chimpanzee. I really loved it. And a great book to develop your kid’s interspecies empathy!

I'd love to know from any resident Julia experts: what are your favorite examples of active, high-quality Julia packages?

And perhaps more importantly, what is missing?

I have high hopes for Julia becoming the defacto open-source scientific language. Despite Python and R both having a massive head start, I'm willing to bet that talented engineers and scientists will be drawn to Julia to implement their next-generation frameworks owing to the powerful features that it offers.

For example, the fact that an array of unions such as Array{Union{Missing,T}} is represented in memory as a much more efficient union of arrays is a perfect example of where a clever compiler can make the logical thing to do also the efficient thing to do!

One of the authors here, happy to answer any questions should anyone have them.

For information on the neural net framework we wrote, check out the 'guide page' that summarizes the main functions:

http://reference.wolfram.com/language/guide/NeuralNetworks.h...

The tutorial collection should give a good sense of how high-level and clean the framework is:

http://reference.wolfram.com/language/tutorial/NeuralNetwork...

> 27,648 Nvidia Tesla V100 GPU modules

holy shit

Any HNers want to share their favorite examples of science tourism?

On a train and don't have great wifi -- but there was a super-cool poster at ICLR which demonstrated that training an RNN to perform dead-reckoning naturally produced grid-like cells. Is this an extension of that work? Or an independent discovery of the same phenomenon?

I'm trying to reproduce that original work, so far without success.

Is it weird that I saw the title of the piece on HN and immediately guessed “Jia Tolentino”. I like her writing too.

Hello! I'm a fan of this latest publication, distill.pub, and your previous work.

I have three questions:

1. Flexible interfaces based on the kinds of grammars you describe in this post feel like their time have come, as an important part of scientific communication, but also as a tool to create, test, and update hypotheses themselves. Entire experiments could be written in the language of these grammars, where transformations and operations directly produce the artifacts that support scientific conclusions: things like means and variances, distributions, statistical tests, scatter plots, and the like. What are your thoughts on that? Is that a direction in which Lucid is interested in going?

2. I'm interested in how to automate the creation of such linked interfaces, whether they be for convnets, language models, seq2seq type models, etc. The idea would be to have the user write a query in a DSL that explicitly connects specific interior activations (on a given axis/axes) with gradients and visualization techniques to produce a linked interface. I haven't yet played around with Lucid -- is Lucid already such a DSL for tensorflow? Or does it rather collect various instantiations of the general grammar that have proved useful?

3. What are the best examples you know of linked interfaces and grammar-like approaches to visualization inspiring important discoveries? Within deep learning, it feels like the slam-dunk example is OpenAI's sentiment neuron, although that was a fairly simple form of visualization. I'd be curious to find more examples, both inside and outside the field of deep learning.

If anyone's curious this function is coming out in Mathematica / Wolfram Desktop 11.3, which is due to be released in the next week or two, and the underlying net will be available from http://resources.wolframcloud.com/NeuralNetRepository/

LIME dates from 2016. For a more recent approach (“anchors”) from the same authors see https://homes.cs.washington.edu/%7Emarcotcr/aaai18.pdf

The problem with Curtis is that he lacks a positive vision of the future.

I regularly run around Molteno reservoir, pictured in the article. It’s been a regular and depressing reminder of the water crisis to see its level titrate down day by day.

Most people here seem to be in denial about the kind of life that awaits. Either people don’t think about it, or believe the official pessimism about the desalination schedule is a scare tactic to delay zero day and hence the cost of the eventual emergency desalinization measures the local government has waiting up its sleeve.

The alternative is unthinkable: a large percentage of the economy of this city is tourism, so imagine what happens when foreign guests cancel their hotel and airbnb bookings en masse.

A few days ago my apartment building installed three massive 5000 liter tanks of water storage. That’s enough to last all the tenants about 10 days at 25l/day/person. If those tanks are filled from the mains it just accelerates zero day for everyone else; if from rain water, well, there’s too little rain at the moment to fill them (and is roof water even potable?). Perhaps the idea is that it’ll be trucked in. No-one knows what is going on.

I suppose we’ll find a way to manage. In a way I’m glad I’m here to experience it, as it’s almost certainly a premonition of things to come in other cities later this century.

Thanks for your blog post!

Are you familiar with http://www.offconvex.org/2016/07/10/embeddingspolysemy/?

I read a blog post maybe a year ago that explained word embeddings under an 'atom interpretation', in which word embeddings are really sparse combinations of 'atoms' (I don't really remember more than that). It was very interesting, but then I forgot about it. Trying to find the post again I only came up with the above paper, which is probably the same idea, but it wasn't what I originally read. Wish I could find it.

This seems like great, quality work. Teaching is an ideal laboratory to explore these ideas, because you're helping students to build mental models with these interfaces, rather than merely aiding developers who already have mental models. Building the models in the first place is the hard thing. But having production tools that would allow experienced developers to more efficiently do this kind of thing is also a great boon.

Introspecting into how I prototype new ideas and algorithms, the kinds of intricate tracing and exploration that are automatically done by Seymour are things I often do "manually", using a cobbled-together set of tools (one of which made it into Mathematica [1]).

Sometimes, because its fairly easy to do in the language, I quickly code up once-off tools to help me verify solutions. For example, I recently finished a new type inference system for neural network layers that allows very flexible 'backwards' and 'sideways' inference of tensor rank and dimensions. Surveying its behavior on all the possible inputs on the various kinds of layers is not feasible via manually typing REPL examples or unit tests. So I ran the inferencer against all possible cases of a certain form, and by visualizing the results as little squares (with various other info behind tooltips and mouseovers), I checked each result very rapidly [2] and soon found cases I hadn't thought of.

One thing I wonder from my experience of doing this kind of thing (I've written various visual debugging and logging frameworks, and code visualizers of all kinds), is whether a prescribed framework and set of workflows is actually preferable to flexible tooling that you can inject in precisely the way you want.

I've often abandoned my visual tools because they are too hard to customize into the exact thing I need to answer a specific, scientific question that comes up during debugging or implementation. Also, when you are already overloaded with thoughts and ideas about what might be going wrong, you kind of want to be able to ask and answer extremely precise questions, rather than swamp yourself in visual data that you have to interpret in light of your hypothesis. Clever presentation can only help the overload so much.

I wonder if what is needed is a kind of "side program" that one could use to annotate the main program in very precise ways and ask questions of its execution, or to produce maps or timelines of aspects of its behavior. But not via interactions with a GUI, although those are useful too, but as actual code that lives alongside the real code, using the same language. Something a bit like a souped-up, visual dtrace, perhaps. I don't know what is already out there like that, but I would love to see work in that direction.

[1] http://reference.wolfram.com/language/ref/Echo.html

It's not really shown well in this documentation (I didn't write the docs), but Echo is very easy to insert because you can just write "Echo @" as a prefix, anywhere in ordinary code, and it basically a no-op. I have a macro system that makes it even easier: the single quote character (which normally means Derivative) can be put next to any expression and does the equivalent of an Echo. Being able to inject and remove these little probes anywhere in the codebase with a single keystroke makes the good old "print debugging" actually pretty effective in a lot of situations. And various forms of cleverness make it more useful, like putting it next to an assignment will also print the variable being assigned.

[2] https://imgur.com/l7XwbCQ

The first two lines of each grid represent the inputs to a CatenateLayer, the last line is the output. Red indicates an invalid shape. Gray is "unknown". The other colors are fixed dimensions like 2 or 3, or dynamic dimensions (used for var-length sequence inputs). This display is from a buggy version of the inference system, don't try to check it!

Just wanted to say, it's always a real treat to watch Mill talks, and I thank Ivan for putting in the hard work of making them so good! (The negativity I see here on HN really disappoints me).

Also, is the thread talk close at hand? I feel I learned less from this talk than usual; most of the material was already discussed in the security talk.

Where do you see a nice list of these types?

Great stuff! I'm a fan of what's gone up on distill so far. Question for colah and co if they're still around: When does the first issue of the journal come out (edit: looks like individual articles just get published when they get published, n/m). Also, that "before/after" visualization of the gradient descent convergence is intriguing -- where's it from?