Experiments in Constraint-Based Graphic Design | Hacker News Reader

Experiments in Constraint-Based Graphic Design

anishathalye.com

211 pointsanishathalye42 comments

Hi HN! I've been frustrated with standard GUI-based design tools, because they don't match the way I think: like a programmer, in terms of relationships and abstractions. (Does this resonate with the community here?)

So I've been hacking on this new DSL for design that allows the designer to specify figures in terms of relationships, which are compiled down to constraints and solved using an SMT solver. This also leads to good support for abstraction, because constraints compose nicely.

The system is still a WIP, but it's usable enough that I made all the figures and diagrams in my latest paper and presentation with it, so I thought I'd write up a blog post about it and share some results.

This post is about how I think about design and how the programming language implements this philosophy. It also contains a number of case studies of designing real figures with the tool.

There are also some (probably useless?) but really amusing results, like what happens if you solve figures via gradient descent: https://www.anishathalye.com/#gradient-descent (the result is oddly satisfying).

I'd love to hear what you all think about the ideas in the post. Do you share any of my frustrations with existing design tools? How do you think when doing design -- does your model match mine, or is it something different? Do you currently use any design tools that you're really happy with, that I should know about?

Are you familiar with "metafont/metapost"?

Metafont was the original system for drawing fonts for TeX and used constraints to describe the character shapes. Metapost extended this to diagrams in general. They might be an interesting point of reference.

This is something I very recently was wondering existed. I was thinking of creating a vector logo for a project, and was curious if vector software was capable of defining relationships like you have done here. It would allow for easily determining what ratios between elements look most visually appealing. I'll definitely take a look at this and let you know if I have any feedback.

Is there actually no other existing tools that have a similar functionality?

I have done a similar tool but based on geometric relations instead of constraints, you can see links in my profile (it's only on mac app store at the moment)

Also Haskell Diagrams.

https://archives.haskell.org/projects.haskell.org/diagrams/

I am curious what sort of technologies and math are used in that kind of program.

All I have to say is "bravo!". Geometric constraint solvers are vastly underused and any vector based drawing tool ought to have it.

I find it interesting that you tend towards SMT solvers instead of gradient descent solvers. Is it because you want to avoid having to specify approximate solution? CAD tools get away with this because the draughtsman is providing the rough starting point.

Also, as a further question? Is it open-source/free software? I couldn't find any links.

Thanks :)

I'm currently using SMT for a number of reasons. One issue is that I've found the constraint is often non-convex, and this sometimes causes issues. And yeah, as you guessed, the initialization is also a hard part. With some figures, I've found that gradient descent can find the right solution with a good initialization (or even with just trying a handful of random initializations), but it doesn't work every time, and I wanted to avoid relying on the user to provide a good starting point.

The right thing moving forward might be some hybrid of SMT and gradient descent, especially if an interactive GUI editor comes into the picture. SMT could provide the initial solution, and gradient descent could help when the user manipulates the figure using the mouse.

A couple years ago, there was a constraint CSS library but the startup behind it went under.

https://gss.github.io/guides/ccss

Do you plan to add a GUI editor too? What do you think of Xcode/Figma's constraint system?

You can use something like https://diez.org/ to provide easy cross platform support. (Note it's only partially open source)

The browser and most "responsive UI" platforms like Flutter have a built-in constraint solver to process layouts. However often the language does not provide sufficiently powerful systems to express constraints in relation to each other, so for the backend you can probably take advantage of existing code when Z3 is too heavy a dependency. (I am not sure if this is possible for all layout solvers, you will probably have to prove it mathematically that the CSS (or whichever platform you are targeting) solver is equivalent or of higher abstraction level in relation to your design language so it can generate the appropriate code. Otherwise the purely declarative approach might not work and you will need JS/Dart/the native scripting language.)

Gradient descent looks exactly like the effect you would get using a verlet-based physics constraint solver. I wonder if there is a deeper similarity between them beyond being iterative?

This looks very nice, I've been a fan of constraint-based systems for some time.

I was surprised not to see any reference to at least some of all the previous work, such as the arc from ThingLab[1] to Cassowary[2] that led to commercial systems such as Apple's Autolayout[3]. Or the Garnet[4]/Amulet[5] systems, with lessons learned[6]. Or Feiner's Drawing with Constraints[7]

You also initially write that the alternative to constraints is to position manually/numerically, but that obviously isn't true, as you have link to the code to do it procedurally later. In the same vein, you initially claim that you can't have abstraction without constraints, but that's also obviously not true when you have procedural abstraction.

You do say that the constraint code is "nicer", and I would probably agree with you, it would probably help to focus on that difference.

[1] https://en.wikipedia.org/wiki/ThingLab

[2] https://en.wikipedia.org/wiki/Cassowary_(software)

[3] https://developer.apple.com/library/archive/documentation/Us...

[4] https://medium.com/@donhopkins/constraints-and-prototypes-in...

[5] https://www.cs.cmu.edu/~amulet/amulet-papers.html

[6] https://www.isi.edu/~szekely/contents/papers/2001/lessons-le...

[7] https://research.cs.wisc.edu/graphics/Papers/Gleicher/CMU/br...

I remember wishing for something like this when I started working in Illustrator after using things like rainmeter and hand-writing SVG code for ages. Of course, I don't have the patience or dedication to implement that sort of thing.

I'll definitely give it a go if I find the time. I love the idea.

If you could make it work with LaTex you could probably find some happy users in IEEE / ACM type communities that would use it for drawing figures in research publications. However, it probably won't take on unless it's made open-source (most researchers are on tight budgets already!)

There once was a "smart" structured drawing program called Aldus Intellidraw

Basalt as it exists today already integrates pretty well with LaTeX -- I have papers written in LaTeX that have Basalt figures. The figures are rendered into SVGs, converted to EPS using Inkscape's CLI, and placed in the paper with \includegraphics.

The integration works pretty well, and it's even possible to make the fonts and sizes match (e.g. see Figure 1 in https://pdos.csail.mit.edu/papers/notary:sosp19.pdf).

I love seeing this, and I was dying to try it out, only to find that there appears to be no such option? :(

This is really cool, and looks appropriate for the hacker crowd.

I'll confess that one of my biggest wishes is for a universal standardized system and visual language+tool for graphic design with constraints -- so that graphic designers can learn a single system and simply export to HTML+CSS, to iOS layout, to whatever it is.

Implementing designs in front-end requires so much back-and-forth with designers -- "what should happen when this text overflows, what should happen if the viewport width doesn't fit this", etc etc etc. I wish this stuff would just be built in to visual design tools, zero code required, and map perfectly to software implementation. It's silly that it's almost 2020 and we still have to manually map this stuff to code.

I'm working on that right now, actually. Though it will be a while before it's ready to see the light of day. There are a couple of challenges.

The first obvious challenge is consistent rendering and performance. The sheer number of targets is overwhelming - browser/ios/android/tv/etc.

The second less obvious challenge is to clearly distinguish what designers are responsible for, vs what developers are responsible for. Some things are easy - "what should the text color be", for example. Other things, like determining accessibility roles, are less obvious.

There needs to be a tool that allows designers and developers to collaborate while maintaining control over their own domain. Not an easy thing to pull off.

Thank you all for the pointers to related projects and research papers!

To address a couple common questions:

Q: Is Basalt open source? A: Not yet, but it will be. The language is still under heavy development, and it's currently being completely rewritten in Racket. I'm waiting until it's a bit more stable to release the code.

Q: Are there plans for a GUI tool? A: Yes, I think it would be awesome to have a GUI tool! It seems like a hard problem to implement this, so it might take some time. The live preview is the best that I have currently.

I love this. I really hope to see the Python version.

I'd love to explore what's possible with Basalt in a language I'm comfortable in, and integrate it with my favorite tooling.

I once took a course on AutoDesk Inventor and was blown away by its parametric sketching tools, which allow you to define constraints on points, lines, and angles. It was very easy to draw a rough sketch of what I wanted and then add the necessary constraints to "clean up" the design—I want these objects horizontally aligned, with identical dimensions, arranged equidistantly from this point, etc.

I've been longing for a 2D diagramming tool with just this feature. Unfortunately there are not many options. Solvespace seemed the most promising last I looked: http://solvespace.com/index.pl

I hope in the future there are GUI tools that can help with designing Basalt documents.

I once took a course on Autodesk Inventor and was blown away by its parametric sketching tools, which allow you to define constraints on points, lines, and angles. It was very easy to draw a rough sketch of what I wanted and then add the necessary constraints to "clean up" the design—I want these objects horizontally aligned, with identical dimensions, arranged equidistantly from this point, etc.

Exactly. That's the right way to do layout, with a constraint-based GUI. The high-end CAD products have had this for well over a decade. The CAD people can even do constraints on curves, like having a line be tangent to a circle or an arc and line meet with equal tangents. This is far beyond what the HTML crowd even aspires to.

The Basalt people, though, have a textural interface for graphic design, which is pounding a screw. Ask any graphic designer.

This is very cool! I hope this gets released--even if it's a rough prototype, I would still use it in its current state.

Like others (apparently), I started writing a system remarkably similar to this earlier this year. I just wanted to generate some simple diagrams programmatically, and went down the rabbit hole of creating a Python library to combine Z3 with SVG generation. For whatever reason, I was getting awful performance from Z3, so I started writing my own constraint solver. That ended up being a lot more of a pain than I had the patience to deal with, so I abandoned it.

> What if the figure design were changed slightly, for example the circle was to be inscribed in the rectangle? With Illustrator, it requires recomputing all the positions by hand; with Basalt, the change is one line of code:

Hi. I've been using Illustrator as my main art tool for most of twenty years.

1. Draw a circle. Give it a stroke in some color.

2. Open the Appearance palette.

3. Using the button at the bottom of the Appearance palette, add a new stroke. Make it another color.

4. Select this stroke and add effect>convert to shape>rectangle, relative sizing, add 0pt in X and Y.

5. Add effect>distort and transform>transform to the stroke. 70% in X and Y.

6. Open the Graphic Style palette and make a new style. Give it a name.

7. Visit the Appearance palette's menu and turn off "new objects have basic appearance".

8. Select the Graphic Style you made, draw some circles using it.

9. Turn off the Transform effect in the Appearance palette; hit 'redefine Graphic Style' and watch every circle change.

While your later examples like "make a little graph constrained to a shield" are more complex to make, there's still ways AI could help you do this stuff - you could maybe make a pattern brush with a circle for the end and corner pieces, and empty space for the main body, pile that on top of a plain stroke, and quickly drag some points around until you had a shape you like.

Eventually your examples get out of the territory of things I'd consider sane to do automatically in the tools AI has. But it can do more than you think.

okay enough procrastinating back to drawing my crazy comic in AI :)

I don't understand neither the author's nor yours approach to doing this in Illustrator.

If you want to have a square inset into a circle in Affinity Designer, you:

- Draw a circle.

- Create a square from the middle of the circle outwards. The square will _constraint_ perfectly to the circle.

If you want to do the reverse (circle inside square), it's even easier since the bounding box of the 2 objects is one and the same ...

I have the feeling that the author is simply unfamiliar with current design programs. Those haven't stagnated like one might think from the article.

It's more about drawing multiple squared circles without hassling with cut-and-paste.

Cut-and-paste isn't a hassle, it's a very ergonomic tool.

I've seen some drawing software once that was particularly nice in that it pasted under your mouse (instead of next to the copied object, or in the middle of your viewport, etc.). Unfortunately, I can't remember which software was that.

I think this is super cool and definitely fits my mental model of how I think about diagrams and designs! Would love to take a look at the code if its available. I'm not a front-end engineer but it seems to me like a more expressive version of CSS parent/child tags and width/height percentages.

Eh hem... One of those developers of constrained "CAD programs, not practical graphic design tools" chiming in: This is cool! Very nice work and beautiful presentation. It makes me pine to build an actual nice website explaining what I did for B-splines/could do for subdivision (so I assert). Or you know, a Rhino plugin, etc. Combining trains of thought, wouldn't constraint programming for B-spline design be applicable to fonts? I do not know anything about fonts really; I just had this belief that font design was often done with splines of some sort.

Wow, this is eerily similar to a thing I've been working on for a few years as well. Very cool! I love seeing other people who are asking the same questions as I am, and arriving at similar conclusions.

> Basalt doesn’t place many restrictions over these equations, which gives great flexibility to the user, but it makes the underlying implementation more challenging because it has to solve these equations automatically.

Given you seem to focus on diagram layout, is the added complexity in defining and solving arbitrary constraints worth the power you get out of having them? Or, can you give an example use of arbitrarily powerful constraints?

This idea is also closely related to Jen Simmons's “Intrinsic Web Design” concept. There is already some automation that can be done on web layouts at the moment.

More about it: http://www.zeldman.com/2018/05/02/transcript-intrinsic-web-d...

You may be interested in Penrose (https://penrose.ink) which generates graphics of math figures from equation. I don’t know if they have as much in the way of a constraint language, but it’s certainly in a similar vein of work, and I know they use some force directed layout.

You had my curiosity but seeing it implemented in Python and so Pythonic now you have my attention.

Figma's recently released Autolayout is somewhat analogous: https://www.figma.com/blog/announcing-auto-layout/

You can deeply nest Autolayout frames.

Very cool. How does it differ from Cassowary?

https://cassowary.readthedocs.io/en/latest/

Cassowary is an incremental solver for linear systems of equations and inequalities, while Basalt's constraints are more expressive (which is why it needs an SMT solver). One example of something that's not expressible in a linear system is computing bounds of a group of objects, which has been a common pattern in my Basalt figure, e.g. to position a group of objects next to another. The Group element in Basalt computes its bounds by computing min/max of its elements' bounds, but Cassowary constraints can't contain min/max.

I'm pretty sure you can implement min/max with a Cassowary-style constraint solver using multiple constraints with priorities.

For example to get the max between v0 and v1 you can use those 3 constraints:

[v0 > limit @ priority1, v1 > limit @ priority1, limit == -infinity @ priority0]

|limit| contains the max, which you can use in other constraints.

Very smart and original approach. I’d love to see a instructional video on how you implemented it in Racket or the sorce code when it goes open source. Will be keeping an eye on it for sure

Interesting you find these problems a lot on CAD design, in 2D you can look at tools like AutoCAD that do similar things in a different setting.

Isn't CSS something that most definitely deserves mention as prior art here?

Stuff like this makes me feel so dumb and grateful that I have a job. lol

Could this also result in a better latex? The layout engine leaves something to be desired.

Reply on news.ycombinator.com