Bret Victor: Seeing Spaces [video]

> purpose-built and general-purpose tooling and environments

That seemed to me to be a big unresolved tension in his comparison between command centers and physical workshops. Command centers still just have lots of general use multi-purpose workstations. TV stations do much better on this point, interestingly.

While they all allow for seeing, I wonder what kinds of improvements we could have with purpose-built HID.

Especially for programming, the tools have to be narrowly tailored to the examples since you're constantly wrestling with the specter of Turing-completeness. Any given program is an instance of an infinite number of more general classes of programs, and it's the tool designer's job to choose which dimensions of the design space are meaningful and important enough to be worth simultaneously visualizing the consequences of possible alternatives. I think it's going to take some very judicious integration of recent work on modularity-enhancing programming paradigms into the design of reflective language implementations before it becomes tractable to build responsive special-purpose reflective tools on top of a generic infrastructure. Or at least that's the strategy I'm trying.

That bias came from the necessity that engineering work is mostly invisible and you can't completely generalize every concept to fit every project. You need to put in extra work to visualize a graphic that would respond to the state of the system and the concepts you are visualizing might only be relevant to the project at hand. Despite these challenges, when we can visualize certain general concepts, which apply to a wide group of systems, but engineers do a good enough job and rarely are visual designers or UX experts involved. At that point its a cultural issue and that's why I'm grateful Bret Victor is around to advocate better designed tools for engineers.

I can't tell if everyone one is like that (I assume not), but I am extremely reliant on visualization, and I suppose Bret is also. I believe profoundly on it's power partly because it's the only way I can really get things done properly, which is why those tools resonate so much to me. I have a friend however that shudders every time I mention making programming more visual.

I mean, If I'm not visualizing something, I can perhaps find some solution in a logical way following some guidelines -- much like following a recipe to solve a equation or doing trial an error to solve an algebraic problem. But critically, I can't create this way. I may stumble unto something useful, but it's an entirely different process from creativity -- it's efficiency is so much lower that it's qualitatively different.

But my difficulty at that is not fundamental, I think. I believe you can be creative purely 'algebraically', but I have no experience with it -- but I do believe that some people have the same efficiency that I have when I think visually and perhaps stumble a little more when trying to visualize things themselves.

I do think this diversity is quite good, but that's missing the point. My point is that Bret's tools shouldn't be universally essential, but are probably universally beneficial. And for some people, like me, they'd be simply enabling.

To give an exaple: when I see a system of linear equations, I don't think of a bunch of steps to solve them and the number of cases that the solutions may look like in terms of constraints of variables an so on. I think of the Image hyperplane, which can be abstracted as a plane in 3D, and the Kernel, which may be for example a line not on the Image. Then instead I wonder if the Kernel is non-null, what funny things this matrix is doing to the vectors (are they rotating, contracting, and so on), or what are the invariant subspaces. I can answer most questions one could ask about such system, but it's a distinct way, I'm not sure if more efficient or not (and that may be up to the task).

I think there's an important misconception about Victor's works and ideas. Augmenting human intellect is not only about visualization, it's about gathering and merging symbolic, interactive and visual representations in a single tool. He brought this point in this conference†, quoting research from Jerome Bruner ††. He shows the example of an electric circuit. Increasing some value on a resistance and seeing the change reverberating on all the plots is equally symbolic, interactive and visual.

† http://worrydream.com/#!/MediaForThinkingTheUnthinkable

†† https://en.wikipedia.org/wiki/Jerome_Bruner

This idealism appears to be one reason why he left Apple:

http://worrydream.com/Apple/

it blows my mind is how little of what he's put forth has led to others creating startups out of them.

...what blows my mind more though is he keeps giving talks and hasn't open sourced any of his software on github, like he said he would. i doubt he's working on some big product that puts all his software to use. it's all going to waste and could have helped propel the visions he's set forth. it makes zero sense.

I just wanted to make a couple of comments; 1. I think that when something is new, like Google Glass or occulus rift, we (as nerds) tend to think it's the future because it hasn't been possible before before we have quite worked out when it's most applicable. Giant displays are awesome when you're brainstorming a topic with people in the room, which is what maker spaces seem to be about.

I can see glass being good for things like surgery, where you need to concentrate your attention on something, but would like some extra data to be easily available.

VR might be a substitute for live brainstorming sessions, and it might allow you to visualise in new ways. But latency is going to be killer across remote locations, and that's a big challenge in terms of infrastructure.

2. I'd imagine as these types of techniques become commonplace, we'll have to learn more stats.

You could also apply clustering and neural nets to a lot of data sets, but yeah - maybe we'll all have to become statisticians.

> 1. It seems, possibly, the exact wrong time to make rooms with giant displays. With things like Google Glass and Oculus Rift as first gen (2nd?) VR/AR you could project all of that info virtually and cheaply and be able to have all the visualization he describes wherever you are, not just at a makerspace that only a few people can use at a time.

I think the reason he wants giant displays is the reason we have giant shared displays in control rooms to: so you can see what others are looking at. Don't forget maker spaces are supposed to be communal spaces where you share ideas. You can learn a lot just by watching someone else use a tool the right way. And then there's of course the collaboration, where shared displays are even more important.

> the system tries every setting and graphs the results so it's easy to pick out the best setting. How would that happen? How does the system know what "good" is?

In a research field of Parameter Tuning we try to answer that question. The field is more focused to optimizing algorithm parameters, but it could be applied to physical world of robots etc., if it is feasible to automatically repeat the experiment few dozen to around hundred times.

If we would do like Bret proposed, by logging each experiment we would already have done some "probing" on the parameter space. This, in turn, would allow us to build a statistical model of the phenomena and then minimize/maximize on that. The resulting parameter configuration would then be evaluated, the model updated and the process repeated until satisfactory level of performance was reached.

See for example the recent works from Hutter et al. [1], where they use Random Forests with parameter tuning to make parameter "goodness" predictions (in order to reduce the actual experiments on the target algoritm/robot/whatever).

[1] http://www.cs.ubc.ca/labs/beta/Projects/SMAC/

I think display are, for now, the better option. AR and VR are still far too in their infancy for them to be as intuitively useful as a room full of displays is to NASA or a power grid operations team(per his example). That will change with time of course, but for something we can do now and over the next few years I think displays and projectors are the way to go. In a way, the projectors over the worktable he shows are(in effect) a limited form of augmented reality.

There are two big problems that I see with the current generation of VR goggles for applications like this. The first is bandwidth, the Oculus's screen had too low of a resolution for densely packed information and text, this may have been solved in the latest but I haven't seen it in person. Google Glass has the problem because of it's really small size, trying to access and manipulate much information on that interface would be extremely slow and cumbersome.

The second is with full VR goggles. Right now there's not a really good way to be able to use them and interact with anything outside the VR environment.

For your other point about the graph the system doesn't have to pick out the best on its own. Just having the possible values and the results displayed the user could pick which one achieves their goals best. For his light following robot though there's a pretty easy way to evaluate the parameters, how far is the robot from the light, that is a simple function and could easily be programmed.

The integration of the whole room and how you get the robot to automatically do many runs with different parameters, for the viewing across possibilities, is where I think a lot of the difficulty lies. To do that with a small robot you have to have a lot of things automated: robot repositioning, light movement, data collection on the robot, etc.

I think it would be some sort of teaching the system what good it. Let's say you are making his little robot. You want it to follow the light. So you pick a variable (light sensor reading) and optimize for a certain setting or direction (this reading as high as possible). The tool track that reading over the course of each try and picks the "best" one. This will have complexities of course. Do you want the highest single reading, or the highest mean or median over the course of the test? etc. But definitely solvable as he states. We managed to build 3d printers and space shuttles, we can build tools like this.

I don't think he was saying that the computer itself decides what the best setting is, but rather that it tests a range of options (given by the user) and presents the results in aggregate (using reductions specified by the user) so that the user can more easily make the decision.

This specification of the ranges also solves the problem of optimizing multiple parameters simultaneously — people usually begin with hunches about what the optima are, which significantly limits the number of possibilities the system will have to test.

The bigger problem I wondered about, is that he glossed over how to "reset" the robot, to start out in a set initial condition for each run.

He had some fade-to-black effect in his presentation, after which the robot magically appears at its starting point ... yeah. You're basically going to need a robot arm if you want to automate that bit.

Interesting. I frequently think back to his talks and use it as a framework to guide my approach especially when debugging. I think about what I can see, and what I can't, and try to address what I can't see. This process has broken through many bugs that seem challenging until they are visible.

> All of those spaces revolve around monitoring

It's not only monitoring because it's not passive, they interact with the system and make changes on the fly, so it's making in a sense too.

I agree. The big-screen control-center examples he gave are for monitoring a mission in real time. The Hadron Collider scientists analyze the data later on ordinary-size monitors. Its the systematic data collection that is important.

Thanks for the excellent link. I only took a quick look, so maybe the information is somewhere on the site, but do you know if this person (or others) are doing similar work for robots in 3d?

no, he's not. this is his weakest presentation yet. he was on, but now he fell off. it's not making any sense that he's gone in this direction, and let last year's software ideas wither away on the vine. He should actually propel forward last year's ideas by at least releasing the stuff he said he'd open source on github. Instead, he hasn't done that, nor has he made his own commercial company (which would be a perfectly understandable route), but instead is giving lower quality talks that regurgitate material about observability that was way more profound when he shared it in relation to software, which by naturel, unlike hardware products, is unobservable.