The brain maps out ideas and memories like spaces | Hacker News Reader

The brain maps out ideas and memories like spaces

quantamagazine.org

252 pointsandyxor72 comments

This is why I think 2D and 3D languages are the future (and of course, are a big part of the present and past, just not in traditional programming languages). Human minds are highly evolved to work in 2D and 3D.

Further reading: 1972 - Mark Wells' A Review of Two-Dimensional Programming Languages

https://github.com/breck7/wells/blob/main/wells1972.pdf

Oh man I'm excited to read the replies to this thread. I'm working on my own 2D language right now as a layer above JS. Let me know if you'd be interested to beta test! I have a sense that the programming community at large will reject it at first, but if it helps non-technical people ship it's a win in my book.

Is it better the befunge? If so, how?

Befunge is very cool, although a bit difficult to onboard users onto :)

My language is a lot simpler: it uses functional programming principles to transform input to output without storing state. The programmer would place down blocks on the screen and draw lines between them to indicate where they want data to flow. They'd start on the left (input side) and end on the right (output side). The cool benefit of this paradigm is that it's recursive; you can build blocks that contain blocks that contain blocks etc... all of which shuttle and transform data from the left to the right.

The other part of this is that it won't just be a language, I'm planning on building an actual cloud environment where you can click a button to deploy your code onto a REST endpoint, so you can integrate it with your existing websites + scripts + apps.

pitch: 'buzz tracker for data transforms.'

https://en.wikipedia.org/wiki/Jeskola_Buzz

(which is admittedly a terrible pitch, unless you're trying to hook an old demo scenester...)

My vision is closer to Yahoo Pipes / LabView / GNU Radio, but yeah I think you've got the right idea!

Hm. You can have that in modern and multiplatform from:

https://warmplace.ru/soft/sunvox/

Oh, yes, sunvox is great, and probably less obscure than the long defunct buzz tracker at this point in history...

That sounds a bit like Pure Data but on the web.

https://puredata.info/

Have a look at NodeRed and VVVV

I would love to beta test such a thing. My number one current frustration with programming languages is my inability to represent core language concepts / objects in a 2D space. There's constant overhead mapping from my internal spatial representation of a program to the linear 1D that text provides.

EDIT: Ahh, I see another comments that this is more of a block based no code kind of thing. I'm not looking to replace my program with purely visual constructs, I'm more interested in representing the file, object, and function layout on a 2D space. Nested folders of files seems so limiting when I want to put particular functions next to each other, and grouped over _here_, while another set of functions should be placed over _there_.

I don't think the greatest power comes from doing away with textual code, I would just prefer that the units which compose a program can be organized, described, and related to each other in a richer way.

For what it's worth, I do plan to support code blocks where you can drop in arbitrary code where you need it, and you can integrate it with the rest of the diagram by defining the input and output types itself. The goal isn't to replace textual programming entirely, but to build an integrated environment where you reach for visual constructs first and drop down into text when necessary.

For your file example, you could have a "File Open" block that takes a filename as input and outputs a "File Reference" datatype, and then wire that reference to various file blocks (File Read / File Seek / File Write / etc) to perform operations on the file. When needed you could wire a reference to an arbitrary code block, perform whatever operations you want on the file using textural programming, and then output the reference to continue putting it through other blocks. Objects (as in JSON objects) could be represented in the same way, twisting and turning throughout your program as they are manipulated. Ditto for databases, file storage (like S3 or hard drives), sensors and their values, etc. As for functions, you'd be able to build your own blocks from existing ones, so you could take a textual function and turn it into a visual block that integrates with the rest of your code.

You could group blocks together in the editor so that they are literally next to each other as you said, but one feature I'm particularly excited about that was inspired by SmallTalk is the Block Search: instead of searching by name, you describe the input and output types of the block, and the editor will search through the global database for any kind of block that matches your description. For example, you could describe a block that takes two strings and outputs one, and you could get a block back like "String Append." For our hypothetical file example, you would just search for blocks that take in File References, and you'd be able to find every block that does anything to a file, including your own.

I know it's hard to imagine, which is why I'm working on building a demo ASAP so I can show and not tell. I saw from your bio that you're interested in STEM edutech so if this doesn't scratch your itch it may still be helpful for students to get started with software a la Scratch! Let me know if you're still interested - no worries if not though!

A benefit of consistently programming since I was a kid is that a lot of programming, especially code architecture, feels very plastic to me. Not quite visualized in a way I could draw, but sort of using the same circuitry as imagining a 3D model of mechanical engineering.

It’s sort of there in my mind and I can see how things are connected, grab stuff and move it around, etc. Super handy for dealing with systems complexity

I thought developing these mantal maps was fairly common? Or maybe it's just me.

yeah I'm not sure if I could work effectively if I wasn't imagining a list interface like an actual list, a hashmap like a pigeonhole cabinet etc. Not literally in the sense of imagining those physical manifestations, but in the sense of relating to objects by their metaphorical interface.

I think it is common for many of us here, but I do think that the inability to construct these kinds of mental maps and manipulate them in one’s mind is one of the reasons that not everyone can code.

interesting, so 2D would be a "matrix language" somewhat similar to Matlab/Octave?

it would be interesting to evolve a language from 2D or 3D to more abstract, higher dimensional representation of objects and topologies, like the cognitive spaces described in the article which operate on sparse vectors in a multidimensional system of coordinates (aka "grid cells").

i'm wondering if there is a language for defining network topologies like that, having some abstractions for JOIN/LINK operators to form & evolve a network [0] and FIND or FIND_SIMILAR operator for traversing that graph (looking for similar objects or clusters of objects), and something that allows to simulate "small world networks" effects, heuristics and shortcuts we see in real world graphs (including neural connectivity and distributed communication we observe in the brain)

[0] Leslie Valiant, Memorization and Association on a Realistic Neural Model https://web.stanford.edu/class/cs379c/archive/2012/suggested...

> interesting, so 2D would be a "matrix language" somewhat similar to Matlab/Octave?

Not only are the data structures 2D/3D, but the language itself is homoiconic.

I'm thinking of things like 2D+ spreadsheets.

Like this from Alan Kay in 1984 (https://pbs.twimg.com/media/E1UAukcUYAAI9lS?format=png&name=...).

A more recent things along these lines that I worked on: https://www.youtube.com/watch?v=0l2QWH-iV3k

Spreadsheets are a specialized spatial DSL. They are phenomenally successful, and have been around for more than 40 years.

Yup. The most successful PL by pretty much 2 OOMs (1B spreadsheet users vs 20M JS programmers). I think the future will be the best ideas from traditional 1D programming languages merged into a spatial hybrid like spreadsheets.

Seems like notebook programming is a trend in this directing.

A purely function language people can understand and use to solve problems they find important without dealing with their IT department. What a subversive notion!

Spreadsheets look 3-dimendional to me, but the first 2 dimensions are just about data (rows/columns), the 3rd dimension is the language itself (cell value), even though you can specify data ranges in the cell value, the language still looks 1D to me

It's not obvious how representing your program in 2D or 3D can help the brain to reconstruct its inner workings in its own version of 2D or 3D, even if it really needs to do this transformation. There might be a subtle difference between a program and a coffee cup.

This resonates well! Is there even a reason to think that a single version of a representation is all the brain works with?

Program have majors business flow, happy paths, that would be rivers of transformations. Then small affluents attach to it as branches appears.

Not so far fetched but I’m not sure you need a new language to support that. You could press a button and get this view. But I think the idea is to build with always this view on sight ? I have no idea.

This is why Powerpoint has been so popular, imho. It can lead to information atrocities but a sensible use of layout can provide a lot of information that the brain can parse much more quickly than text, and directly provides a spacial layout for recalling it later as a mental model (for example, if you paint subsequent phases as actual boxes, instead of describing how something follows something).

The idea also rhymes a lot with George Lakoff theory that we build concepts as towers of analogies and metaphors, whose bases are quite instinctive and frequently spacial (high is good, broad space gives you options, etc.)

I think they are already. I don't see a file with code as linear text but as a shape. And several files are the other dimension.

And I think not everybody is like this, though! I think there's a difference between "visual programmers" and "text programmers", and this is what causes the flamewars between people who want to "remove noise" by getting rid of parens, braces and semicolons, and people like me who use them as a sort of "signage".

The real problem is that I bet the "text programmers" also write more, well, normal text, and are overrepresented in blogs, articles, and whatnot. (Even though they're wrong XD)

> between people who want to "remove noise" by getting rid of parens, braces and semicolons, and people like me who use them as a sort of "signage"

I'm not sure this characterization appropriately divides "text" from "visual" programmers. The argument I usually see for removing symbols is that the general shape and flow of your program should be able to stand on its own and that minutiae like a misplaced semicolon shouldn't be allowed to play tricks on your eyes -- it's not that the visuals are unimportant, but that they _are_ important and that too many extra symbols lead to hard-to-parse visual clutter and more easily facilitate deceptive control flow.

Whether that argument has any merit or not is its own debate of course.

Programs are already cyclic trees. It's just that code is a somewhat cumbersome representation. Or rather, it's not a very natural one.

At first I actually thought you meant human languages, which are already 3d to a large extent, because they're full of movement metaphors.

A cyclic tree is a graph, fwiw.

A cyclic tree is a cyclic directed graph.

Is python not sufficiently 2D? You could see it as characters in a grid.

It only uses a tiny bit of 2D (the indenting of parent child nodes). Most of it is still 1D (parens, quotes, etc).

Python programs to not copy/paste well into spreadsheets (a simple test for "2Dness").

I think it will be interesting to see what develops as we learn more and more about the mechanisms of our brain and how they intersect with the subjective experience of our mind. I feel reasonably certain we’re going to be able to unlock new capabilities that were previously either undiscovered or not reliably available to humans. Things like the ability for some people to essentially be a human calculator, and others to have photographic memories.

The possible downside however is that we will likely be able to quantify the capabilities of any particular brain based solely on an assessment of its physical attributes. That seems like a dark place.

Couldn't agree more if I tried. The number of hours I've spent thinking about this is ridiculous.

Here is a fun rabbit hole: https://www.youtube.com/results?search_query=minicolumn&sp=E...

Reading this comment is like what back mirror was before it went down hill.

Or we might discover that brains are not so fundamentally different and most of the differences arise from the environment? It's the centuries old nature vs nurture debate. Perhaps that is the wrong question to ask.

I agree both extremes are possible. I don't feel confident betting on either extreme. I would bet it's both at the same time (nature AND nurture) to the point where the individual influence of each is practically impossible to disentangle.

I'm glad to see Jeff Hawkins was mentioned. Much recommend A Thousand Brains: A New Theory of Intelligence.

+1. And if you haven't read it, On Intelligence is also a great read.

There is a company that takes this concept and lets you take notes in "spaces".

https://www.nototo.app/

In some ways, Panorama Tab Groups in Firefox is similar

We've been taking advantage of this for at least 2500 years. Our languages show this ("in the first place, in the second place", in Latin, in Greek, ...)

What blows my mind is the concept of sensor fusion and how it works in the brain.

How you can infer the shape of an object by progressively touching it with your fingers even if you cannot see it. Try to implement an algorithm or ML solution that does this and you'll know how insanely hard this problem is.

Could you elaborate on this a bit? I don't have familiarity with that class of problems, so I'm having trouble understanding the obstacles.

My naive instinct is that you would detect and store points of contact between the sensor surface and the object while also positionally tracking the sensor in 3D space relative to an origin. You would certainly need a lot of precision in movement tracking to minimize error. Maybe that's where it gets tricky?

Also worth noting that the problem feels somewhat moot since there are easier solutions (e.g. lidar).

Lidar is pretty expensive in terms of energy consumption.

I adore this magazine, it consistently makes me happy

Might be time to dust off the old memory palace technique

My first thought too. I've always meant to give them a more serious go...

Wouldn't be surprising since spatial memory would be one if not the first recalling system for survival.. piggybacking on it for abstraction would be structurally efficient.

One of the ancient Greek craft of mnemonics is to spatialize what you wish to remember. For example, use the rooms of a house to contain sequential items of a speech.

That's probably why so called memory palaces work

Place Cell/Grid Cells/Head Cells/etc... might be not mapping space but cortex location (i.e., the "where" in the hippocampal complex isn't referring to physical space but from/to cortex location the "when" is recalled or should be stored). Rodent specific navigation abilities might be driving us toward the wrong interpretation of HC complex?

I'm always super skeptical when I see a blanket statement like this about how "the brain" works. Maybe many brains do but there are too many incredibly fundamental variations of how peoples' brains work to generalise too much. Aphantasia, variations in inner monologues, different sensory processing... we really are all our own unique creatures.

But saying "no we are all unique" is another blanket statement.

It's more likely we almost all share the same feature set. Doesn't mean exceptions or tendencies don't exist. You know... Like... Most of us have fingers, ears, skin, etc.

True, most of us share a bunch of processing and the same kind of tricks work on us. And we all share a lot of lower level stuff with most mammals. There's still way more variation between humans mentally than physically.

Reminds me of the Queen’s Gambit where the protagonist plays chess on a hallucinated chess board projected on the ceiling.

I believe that people with hyperphantasia can actually visualize chess boards on ceilings. As somebody who cannot actually visualize things with my mind’s eye (aphantasia) I cannot comprehend this.

You can train your visualization. Check out image streaming, it's a sort of meditation where you focus on what you see when you close your eyes. If you have aphantasia you will still see the purples and blues and green and stars and dots and whatever happens when you close your eyes.

What you do is you speak out loud what you see "A purple blob in the middle, it's moving to the right, now it's morphing into a more green blob" etc

If you do this for a couple of minutes what you will see in your closed eyes will become more shape-like "the blob is now more circular. It's back to a blob again, now it sort of resembles a weird pillow"

Somehow, the feedback loop of objectively describing out loud what you're seeing trains the brain to fantasize more real shapes and objects.

Try it out if this sounds interesting to you!

Wow, that’s extraordinary. Do you think it’d be fun for kids too?

I have no clue, I have personally tried it and yes it can be very fun. It's supposed to be done with two people ideally^1

But it can get really dark as well. I have done this between 10-20 times on my own, and the one time the visuals are about like... fruit, and the other time they're about really really bad super shameful and traumatic personal experiences.

It's not necessarily a game, although I can imagine it has the potential to be one. The danger is that this requires you to speak up about anything that comes to mind, and sometimes your mind will wander to very serious memories or pictures.

I can say that it works incredibly well though. The purpose of image streaming is that it should improve your ability to imagine shaped in your head, and it works _really, really well_. I personally stopped because I didn't feel comfortable speaking out loud many of the things I saw (even when I was alone), it involves "confronting your shadow" in Jungian terms

Here's the inventor's article on it by the way: http://www.winwenger.com/imstream.htm

^1: the person who invented it says it has something to do with how the feedback loop is more real if there is real consequence to what you say iirc

I wonder if there's a correlation between aphantasia and not reading or hearing stories as a kid.

It somehow always irks me when VCs talk about "spaces" in reference to markets and how "spaces" are "crowded", and how that limits their thought into what's possible.

To me a market isn't a space that can be occupied in the same sense of occupying a finite physical space.

Why not? Obviously not in exactly the same sense but if there is a limited resource (attention from customers to solve their particular issue, and dollars dedicated to solving it), and a spectrum of solutions with niches for different clients, the analogy with space seems legit. And if there is a space of solutions and you are very close to another one, and you can only differenciate in price, you can say you are uncomfortably close to the other one. If moving away from the similar product to differenciate yourself makes you very quickly uncomfortably close to another product, then its crowded. Sounds very useful for discussion.

In many markets startups are very far from that being a limited resource. For many startups, to think that the market is a limited space (rather than an expandable balloon) is a big fallacy IMO.

Yeah but then it’s not that the space analogy isn’t apt, it’s just it’s not crowded space. Space is a big thing, it holds entire galaxies :)

Whoever says this is falling into a "limited resources" type of fallacy and is stuck in some form of local-maximum thinking. It's one of the big failures of economics that it treats resources as "finite". There's zero evidence that anything is really finite anywhere in nature. Why should it be finite in economics or finance?

Providing crackpot, pseudo-philosophical examples of why something can't be done can make me look smart but ultimately has no utility.

Ok well what happens when the entorhinal cortex is excised or probed?

Surprised the article mentions Kant but omits Bachelard, topoanalysis and the Poetics of Space.

This feels oversold. As someone that is not visual in thinking at all, I find it hard to even know what this means. :(

How would you classify or describe your type of thinking?

Hard to say. I remember things, but I can't and don't visualize things. I learned the term aphantasia a while back and that hits me fairly well.

It is a lot like a constant word association game. Just sometimes without words. And no narrator voice at all.

Reply on news.ycombinator.com