Create system diagrams with Go | Hacker News Reader

Create system diagrams with Go

github.com

247 pointsngaut63 comments

Go-Diagrams is a Golang port of diagrams [1], an Internal DSL for Python. Diagrams (the name of a software project) models cloud IaaS resources as network graphs and exports DOT [2] files that can be processed by Graphviz compatible tools.

Neither Python nor Go are ideal languages for implementing an Internal DSL, compared to say Ruby or Groovy, so the first question is why an Internal DSL vs. the DOT language (or another External DSL) and the second question is whether Python or Go are expressive enough for this use case? I think the answer in this case is that any language that supports Class-like structures can provide a namespace to describe the common resources, relationships, and constraints found in cloud IaaS.

I'm assuming that the DOT language does not have the facilities to add domain specific names and constraints so an Internal DSL is a perfectly reasonable approach even if it generates intermediate DOT files.

[1] https://github.com/mingrammer/diagrams

[2] https://en.wikipedia.org/wiki/DOT_(graph_description_languag...

I'm not sure if you've ever tried to maintain diagarms in plantuml or dot, but it becomes difficult over time to keep them up to date.

I haven't used Python or Go diagrams but I have been a big user of plantuml for over a decade now. Off the top of my head, here are the main logistical problems with it:

  * It sits on its own
  * Hard to "share" metadata
  * Difficult to manipulate once put into the DSL
  * PlantUML has 12 different languages in one tool

I took a look at Python diagrams and a light bulb went off. Something that would make it so much easier for _everyone_ to contribute. Most people don't want to use PlantUML. But a DSL in Python? Could pay dividends.

I will seriously consider Python diagrams for my next project.

Just curious, do you always create a new HN account to comment?

Simple DSLs are sort of Haskell's bread and butter, I'm actually somewhat surprised there isn't a decent flowchart DSL in Haskell (I know about https://diagrams.github.io/gallery.html, but it didn't seem straightforward enough to get a simple flowchart going).

For someone who doesn’t care language, what’s the difference between this and the original diagrams tool?

It's written in GO! :P

So maybe this means it’s easier to integrate in a Kubernetes workflow? Not sure though.

Is this tool beyond creating diagrams? If not, I don’t grasp the point of porting, writing the diagram is like scripting, for which python is arguably a better choice.

I hope this isn’t a trend similar to Atwood's Law: “Any application that can be written in JavaScript, will eventually be written in JavaScript.”

It is, and with "written in Go" being part of the advertisement.

Its a funny law but ultimately people like using the tools and languages they know. And most languages are good. So id expect youd see most things ported to most languages excepting where they are prohibitivley expensive. Also making things is fun.

I've been using some nodejs applications as tools to deploy other applications (for example serverless) and while it works it is the most trouble causing frameworks.

It absolutely has to created node_modules in center of your source code, and populate it with 120MB of files. If you want to just package it somehow so to make it more like cli tool (like you could do with awscli for example) then tough luck it will break in many weird ways.

It's like the nodejs people threw out everything we learned in decades and are reinventing the wheel. Yes it works great if you have your own project in a single directory and working on it, but any other use case is a horrible experience.

I probably wouldn't use this tool often enough to memorize the API, in which case the type checking and autocomplete that a typed language like Go enables could come in handy

As a PHP developer, I assume a single Go binary able to be used in CI task or simple document scripting tasks is a huge bonus if I don't want my scripts to be dependant on Python and/or PIP.

Every time I have to create a diagram I struggle. If I want something quick and dirty the options are out there. But if I want to create a diagram for a serious project, I have trouble. Ideally, I would love a solution that integrated with templates and allowed me to define my diagrams at a "higher-level". It should also integrate with Latex to look consistent. Does anyone know of a better alternative to TikZ for my use case?

> It should also integrate with Latex to look consistent

How does latex determine how the pngs you include should look like? Or are you worried the uml diagrams look too old?

Edit: Added those parts back.

I'm using Lucidchart to create uml diagrams, which I export as pngs and embed in latex. Costs a couple dollars. I like Lucidchart more for the freeform diagrams where I want to position everything to look pleasing

Then theres yuml, which I render to svg and then with inkscape to png. I used yuml for a sequence diagram because positioning each one manually felt unnecessary.

> How does latex determine how the pngs you include should look like?

In my comment integration refers to text selection and font rendering. As much as I find TikZ too "cloying", I have to admit the end result looks terrific. However, I would settle for anything that improves my current setup. Next time I'll give a try to Lucidchart (by the way, I saw your previous comment). I currently do embed pngs because I find TikZ too verbose.

I initially thought TikZ being about any sort of diagram, so uml did not fit :)

With Lucidchart any text can be highlighted with color independently and there are lots of fonts to choose from. I like it for uml diagrams.

PlantUML may be of interest for some kinds of diagrams.

https://plantuml.com/

Looks promising! Especially the Activity Diagram. Although these diagrams don't look too modern -that base style could see a refresh- it would do the trick just fine for me!

I have gotten really used to drawing all my technical diagrams with Inkscape (system arch, sequence diagrams, product roadmaps, etc).

Granted, it's labor intensive and hard to justify the learning curve just for technical diagrams but I find that once you have built the muscle memory and a good mental map, it's not that slow (and encourages concision and good abstraction to make the diagram simpler) and superior to auto-generated layouts because it gives me the flexibility to make prominent what deserves to be prominent (objects, paths) and complete control over styling. I also hate it when adding a single element might change the auto-optimized layout entirely.

I wonder what folks who use (go-)diagrams at scale think of this. Maybe my solution is untenable for large enough projects.

One thing I like about textual diagramming tools (in code like this project or in a markup dsl) is I can version control and diff changes.

Fair! Do you build workflows around the diffs themselves (collaborate, review, update)? I can already put my svgs in version control and have history/versions. But I've never felt compelled to look at a textual diff of viz code.

Yes, absolutely.

SVG is lower level than I'd want to do this with though - instead of comparing things like individual paths and fill colors, a dsl at the right abstraction level lets you compare things like adding a server to a load balancer in a network topology diagram.

I've made diagrams for my talks in both style (with code and with Inkscape or, these days, Lucidchart). Writing code to generate diagrams had a much larger setup cost but once I had it going, iterating on the diagrams became much easier.

I had a talk about radix trees[1] where I wrote some Haskell code that could take trees and generate diagrams from them. Getting the code right in the first place took a few hours but in this case it was clearly worth it in hindsight for two reasons:

1. I ended up with a lot of tree diagrams. Once the code was working, the marginal cost of adding an extra diagram went way down and I think that improved the talk as a whole.

2. Code made it much easier to iterate on my examples. I had an extended example[2] that I changed several times; if I had had to manually redo the diagrams for it each time, I would not have bothered.

These days, this trade-off (up-front vs ongoing costs) makes the decision reasonably easy. If I'm going to be making a single diagram and I don't expect to change it much, I'll do it manually; if I'm going to be making a series of similar or related diagrams or if I expect to iterate on the diagrams a lot, I'll write some code for them. So far, I haven't regretted following this approach in either case :).

[1]: https://jelv.is/talks/lambda-world-2018/slides.html#/sec-tit...

[2]: https://jelv.is/talks/lambda-world-2018/slides.html#/slide-o...

Thanks for this! Ceating multiple slightly different diagrams manually is indeed a painpoint.

This is interesting. I am working on a tool (sourcespy.com) to create diagrams specifically because maintaining large diagrams in Inkscape and Lucidchart is cumbersome. Another pain is grouping the nodes. I wish there was something interactive where care about semantics and tool takes care of node arrangement.

The result looks nice. I think this type of diagramming need to be as straightforward as possible, with as little typing as possible. I see you use many different types. For example

gcp.Database.Sql(diagram.NodeLabel("Database"))

What does the NodeLabel type provide that you would not gain with just the string?

Looks like NodeLabel is a function which allows you to instantiate a NodeOptions with only the label attribute set in a shorthand way. So the functions taking the results of NodeLabel calls are able to accept more complicated options, the examples just don’t show this.

It's an implementation of the functional option pattern used a lot in go code. See https://dave.cheney.net/2014/10/17/functional-options-for-fr... for a rundown.

It's a functor, though the type system doesn't let you write a reusable fmap.

Great! I am a fan of seeing diagrams being generated instead of manually drawn.

I was a bit disappointed however to find out the project uses Graphviz/DOT under the hood.

What would be the reason not to try a newer rendering engine such as as the Eclipse Layout Kernel (ELK)?

Curious, what's wrong with graphviz?

I've used it for auto generated diagrams of our code. Partly because it's what I know, partly because, surprisingly, there isn't anything free similar.

It's crashy for moderate sized graphs. And doesn't have quite enough layout and styling features. It's good enough but I'm on the lookout for something better.

Have you engaged the graphviz team? Maybe the don't have graphs of that size as part of their test suite, and if you could donate some samples, they would fix the "crashy" behaviour. No software should ever be "crashy" - it's buggy and bugs need to be corrected.

I've come across the same issues that others have reported. The answers have more or less been "contributions welcome". They have every right to say that!

I was thinking about writing a markdown extension that would render a code block into an image using this and this extra system dependency is also what turned me off. It's just an extra roadblock for users and whole block of extra instructions for each platform that users have to figure out. It would be better if it would just work out of the box.

Your extension idea sounds similar to this that was posted yesterday:

Pikchr: Documentation https://pikchr.org/home/doc/trunk/homepage.md

Pikchr – PIC-like markup language for diagrams in technical documentation | Hacker News https://news.ycombinator.com/item?id=24601971

I don't know if you're referring to Go or GraphViz as the extra dependency (or both). It looks like Pikchr doesn't have any external dependencies, it's just a C library.

Hey that’s cool, I didn’t know about this. But there are a few things that would make this difficult to use:

1. How do you even call a Java library from Go?

2. Graphviz generates diagrams, whereas according to the ELK page:

> Note that ELK itself doesn't render the drawing but only computes positions (and possibly dimensions) for the diagram elements.

It also wasn’t really clear to me from glancing at the page, what this does better than graphviz.

Thanks for taking the time to reply to my comment and sorry for the short reply from my side - I am in a hurry.

In case you were looking for a similar text-based integration of a layout engine with your tool there is a textual DSL for ELK called ELKT:

http://rtsys.informatik.uni-kiel.de/elklive/

One could certainly feed that into a binary running the kernel similarly to how a DOT file is fed into graphviz.

Also worth mentioning, the Sprotty project by Mirko.

https://github.com/eclipse/sprotty

This integrates with XText, ELK and the VS Code LSP and allows you to generate diagrams for your own DSL!

Now if this could read something like say Terraform and be able to (if all your infra was defined in terraform) keep infrastructure diagrams in sync with production instead of the idealized thing we think they are that’d be reallly cool.

Any chance you can provide an example of Terraform source and visualization? I'm interested in it for my own project. Thanks!

Yeah that’s the thing I’m looking for too

Very nice, but IMHO this should be the engine behind a file that could easily be parsed such as YML / JSON.

I really don't see the point in having to write a Go snippet to describe the architecture: in the end you're providing only data as an input.

I guess I'll try to do what I just described because I feel like this could have a lot of potential for my system diagrams

But with Go you get a schema of sorts. And also, you can easier programmatically generate "dynamic diagrams". For example by letting it parse a YAML file :-)

yaml is a serialization format. You need at least a schema at which point you might as well write in a nicer language.

I agree with OP on this, YAML seems better suited for such a tool as you can easily add some form validation to it and then serialize it into Go data structures. You really shouldn't have to recompile your codebase in order to modify a diagram.

You don’t recompile the codebase, you build the diagram. You can build it with a tool that converts yaml into svg or golang into svg. The point is that there isn’t much difference as the diagram build process has the exact same high level steps in either case.

> I really don't see the point in having to write a Go snippet to describe the architecture: in the end you're providing only data as an input.

Writing a program to generate the diagram is similar to writing a program to generate your infrastructure which is exactly what you do in "infrastructure as code". If your infrastructure is simple then you may not need infrastructure as code. The same can be said for diagrams but at some point you might want the flexibility if the "diagram as code" approach.

It all depends on the context, you're right.

I was more thinking about my use case without considering all of the possible use cases such as Infrastructure as Code, which this project seems more suited for.

My use case is to create simple explanatory diagrams about a concept / idea rather than a real-world scenario: in those cases, I always end up using Draw.io. This causes me to waste a lot of time dragging and dropping around just to illustrate something simple :(

I guess that with my solution (which btw I'm developing here: [0]), you get a versionable diagram from a YML file that can be parsed and exported everywhere thanks to GraphViz. My idea of the input YML file is the following: [1]. This is a YML representation of the architecture shown in the README.md.

[0]: https://github.com/denysvitali/go-diagrams/tree/feature/pars...

[1]: https://github.com/denysvitali/go-diagrams/blob/feature/pars...

Congratulations on a neat project!

I just forked the project linked on this article, I didn't create the original version myself: my addition to what has been posted here is the idea of parsing a YAML file and getting back the diagram, without having to write a single line of Go code (:

Your JSON/YAML representation doesn't have to be in the go-diagrams module itself. It is just a layer you can build on top of go-diagrams. So I suggest to keep your project independent.

Writing my masters thesis, where i worked a lot with TikZ, the real power came from being able to defined your own graphing primitives.

With that you can defined your own graphing system. Maybe you have a type of arrow that means something UML doesn't have, just define that class and then defined how it will look. Maybe you have a connection type that requires separate arrows in both directions, just defined the syntax.

You can make all these things happen with regular TikZ, but the ability to define macros and classes makes for easier graph drawing, and a unified definition. It also means you can distribute these styles and let others work on your graph language.

Given that this uses Graphviz, you may as well just write you diagram using that if you want something declarative.

Without the added value of good quality PNGs that you'll have to add yourself (:

I love the simplicity for K8s diagrams, this is a neat tool.

package main import ( "log"

"github.com/blushft/go-diagrams/diagram" "github.com/blushft/go-diagrams/nodes/k8s" )

func main() { d, err := diagram.New( diagram.Label("Kubernetes"), diagram.Filename("k8s"), diagram.Direction("TB"), ) if err != nil { log.Fatal(err) }

ingress := k8s.Network.Ing(diagram.NodeLabel("nginx")) svc := k8s.Network.Svc(diagram.NodeLabel("http"))

d.Connect(ingress, svc)

g := diagram.NewGroup("pods").Label("Deployment").Connect(svc, k8s.Compute.Pod(diagram.NodeLabel("web server")))

d.Group(g)

if err := d.Render(); err != nil { log.Fatal(err) } }

What's the benefit of coding a visual diagram over drawing it?

I was at first thinking of automated verification rules that force your diagram to commute over a set of rules you specify but even drawing tools can specify this.

Any people who diagram a lot have an opinion on this?

It depends. When I have a good idea of what I want to draw, and I want to put it in a document, I like to use TiKZ in LaTeX because it is fairly easy (depending on the domain) to produce something pretty that is consistent with the style of the document and easy to change if the surrounding text is edited. The overhead of going through an external diagramming tool is often not worth it in this case, unless it's a quick one-off note.

If I want to get an overview of a data structure, I like to write code that generates dot, which I then render using graphviz. I have successfully used that for debugging a few times.

When I use diagrams as a tool for thinking, I prefer paper and pencil, or, if I need to make many changes, yEd. I also use the latter for system diagrams (I prefer loosely following the C4 ontology) or for quick dissemination of ideas to colleagues when we can't be in front of the same physical whiteboard.

The only text-based diagramming tool that I use for thinking is plantUML, although I have only used it for sequence diagrams.

I don't have a lot of experience with using embedded DSLs for diagramming. I have used 'diagrams' in Haskell which is nice, although knowing the host language didn't mean that it was faster to learn the DSL compared to, say, graphviz or plantUML. It doesn't take a lot of time to learn the syntax of these things, it's all about learning how the primitives work and how they can be combined. An EDSL is mostly convenient if you need to programmatically generate the diagrams from data structures, since you can call the diagram library directly and avoid icky string interpolation.

> What's the benefit of coding a visual diagram over drawing it?

TLDR: You want to communicate abstract relationships with a picture. How the picture is drawn usually isn't as important.

For example, say you want to visualize a graph (the nodes and edges kind). You would like to tell the computer:

1. Here's the definition of my graph.

2. Draw it in a clear, visually appealing way.

Point 2 encompasses things like spacing, layout, color, etc. I would much prefer an algorithm to make those decisions for me, assuming it's decent. They're boring implementation detail.

That said, imo Go is a poor choice for specifying such abstract relationships because of its verbosity. I'd rather use dot directly, or a more concise DSL.

That’s great if it works. I’ve yet to see a tool give me the same diagram layout I as a human can work out in roughly 20 minutes of trial and error. Not to say tools with auto layout aren’t helpful, but most of the diagram DSLs I’ve used have the ability to add constraints that will affect diagram layout which generally accomplish the same thing: have the human intervene in the diagram layout to create a better layout.

Note I’m not even talking about colours and spacing, I know computers can do a better job at that. I’m simply talking about how the graph is laid out and the decisions humans can make about what is still easy to read or which lines are more important than the other lines. Sometimes — often — the layout algorithm isn’t given all the information, the human doing the layout can re-interpret the graph data in real-time.

The best layout system, I imagine, is one that makes layout even easier for humans to control—but still puts the human in full, graphical control, at least as a form of data-entry. The problem I often have with visual tools is they make it hard to “add to the diagram” in a way that feels natural. I often wish I had a command that would push multiple objects out of the way, or a command to align some boxes to a grid such that I could add more to a “list”. I also wish I had a tool that would change the diagram as I zoom in and out, for diagrams that need progressively more detail, and so on. Maybe Miro or similar would do the trick, but then I’ve less visual control.

I think there’s a lot of room for improvement in diagramming tools but there’s also a steep learning curve perhaps. The easiest tool I’ve found to use interactively, personally, is OmniGraffle and I’ve heard Visio has some of what I’m looking for, yEd is nice too, but ultimately each tool has its limits, often because they’re designed around building one graph or diagram disconnected from the rest. The Brain or WebBrain is a nice data graphing tool but isn’t designed around producing diagrams. Often the beat we can do, it seems, is embed multiple diagrams within a document or to use layers to show and hide more detail manually, but then those layers need to me maintained manually also.

> What's the benefit of coding a visual diagram over drawing it?

I'd think the biggest benefit is that it would be easier to reproduce the diagram, iterate on it and save revisions of it in standard tools. It would also allow flexibility in that reproduction which would be super handy if you were creating a lot of similar diagrams.

And the most obvious one is simply you don't have to draw it. Some people find drawing diagrams to be extremely tedious but still need to do it frequently.

If you have "infrastructure as code" you could as well have diagrams of the infrastructure generated from that code too and go-diagrams is one brick to build that.

Reply on news.ycombinator.com