Graphviz: Open-source graph visualization software
graphviz.org
graphviz.org
Graphviz, at its core, is a simple graph description language (DOT), which allows progressive enhancement, and a bunch of auto-layout tools where some double as renderers.
The double-duty of the layout/renderers masks GraphViz's true power, which is that you can create a pipeline modifying your graph source (confusingly referred to as '.dot' files), progressively adding attributes.
This is where gvpr comes in. While most are familiar with GV's all-in-one layout+rendering tools like `dot` or `neato`, gvpr is a generic Awk-like programming tool that iterates over nodes and edges. With it, you can implement any layout algorithm you like (or not!), to specify part or all of the attributes of your graph (like positioning and style), to then be rendered using `dot` to the format of your choice.
Neo4J is interesting. It does something important, mainly, rationalize the mess of graph representations and algorithms. It's not hard to understand why that is valuable, but Neo4J sell it as being "efficient for linked structures" (relational databases don't have foreign keys and indices) and "intuitive" (have you ever tried to read the code for graph matching in Cymbal? I rest my case.) They have investments of hundreds of millions? Worth billions? They contributed something valuable to the tech economy.
It's surprising the conventional relational database products don't pay much attention to this opportunity, but they have gone vertical, e.g. credit card processing, because in the end it's more profitable to solve big end customer problems.
The most complex thing I've done with it [1]: a tool (MIT-license) that builds diagrams of the data and addressing pipeline for a DSP processor, and lets one 'scrub through' the assembler code frame by frame and see the values propagate through the blocks.
Also PlantUML [2] uses it for most diagrams.
Getting layout and positioning the way you want can be tricky but is usually achievable with patience and hidden objects.
[1] https://github.com/paphillips/DFB [2] https://plantuml.com/graphviz-dot
1. Draw "sketch" style graphs using https://sketchviz.com/new
2. Draw graphs of memory usage in golang programs (using pprof): https://graphviz.org/Gallery/directed/pprof.html
3. Embed graphviz into Sphinx docs (shameless self-promotion: https://docs.robusta.dev/master/catalog/triggers/index.html#...)
I'm amazed that this hasn't been addressed. It's been there for years.
Makes the end products impracticable for distribution in a professional setting. It's Ok if you're the only consumer ... not great for other cases.
The relevant code starts around https://gitlab.com/graphviz/graphviz/-/blob/main/lib/common/... If this isn't right, maybe somebody can figure out why. Perhaps the loop stops sometimes on the wrong side of the boundary? Anyway, it's equally possible that when the endpoint coord is handed off to a lower level driver, the arrowhead mitering is wrong. There is no question there was once upon time explicit code to try to cope with this problem, at least in the native PSgen, but I can't find it now. Maybe it wore away as waves of open source development washed over it, along with static tables for a bunch of "standard" PS fonts.
For more background, see for example https://developer.mozilla.org/en-US/docs/Web/SVG/Attribute/s...
In the worst case, this has to be debugged for all the drivers in https://gitlab.com/graphviz/graphviz/-/tree/main/plugin
No doubt there are more details if anyone can remember them.
If nothing like that exists, I can see how it would be a very troublesome refactor.
I'm talking by heart and I could be wrong in any case.
It became a wedding present for one rather central "node" with the title "drosophila neural regulation network" or something like it. No names, just lines and circles. It's still the centerpiece in his home office.
1) is there an option to make the graph horizontal instead of vertical? 2) can you make the svg option separate, instead of hidden behind the PNG button (with CTRL key)? 3) how did you do the handwriting style?
See in the docs: https://graphviz.org/docs/attrs/rankdir/
In the sketch example:
digraph G {
rankdir="LR";
graph [fontname = "Handlee"];
...Super intuitive, I know! :)
https://observablehq.com/@observablehq/dot
Here’s a recent example where I used it to figure out why certain dependencies are in my node_modules:
two steps above the abandoned version, before the web threads attempt, works quite nicely.
They're really compelling but oddly unpopular.
Imo, they're useful for one-off pattern discovery, and they're most valuable for finding single or a few exceptions and outliers in normalized data, and you need to be in an environment where there are asymmetric returns on finding those. Surveillance seems to be their default use case, with open ended scientific research a close second. This comes up with graph based recommendation engines, which are essentially a surveillance/marketing product based on preferences. Most businesses aren't based on discovery of anything other than customers for a transaction they already have.
These aren't problems engineers typically solve, which are more about scaling and optimizing, they're more marketing and sales problems, where you're looking for exceptions and opportunities. (security and privacy are the complementary antithesis of these) A graph based product (imo) is ideal for product marketing analysts optimizing for customer preferences and discovery.
From a product perspective, graphs are analogous to ML, where you'd use a clustering algorithm on loosely structured data to yield categories, comparisons, and implied relationships, whereas a graph yields the same thing over the structured normalized data you can feed into it once you have imagined an ontology to fit it.
Did I understand it correctly that you use it to discover patterns? Are these patterns discovered by just using the layout engines? Arent' other tools, e.g. networkx in Python or cytoscape in javascript easier to use interactively in a REPL? What is the typical workflow (maybe plot, find interesting pattern, change query/data in a loop)?
I'm really interested in how Graphviz can be so great. I am currently working with the other mentioned tools for visualization purposes.
One example of clients taking months is mapping counterparties to agreements. Let's say you have inherited a division that has file share full of contracts and you want to understand the line of business. You get the counterparties out of the contracts and find all the paths for obligations between entities within the division and their counterparties. The graphviz/dot layout gives you a map of all those parties in a single slide and shows clusters, instead of a 3 lb. document with a paragraph for each of them that would have cost a massive amount of consulting time, or interviewing several people to get their narrative understanding of how the business worked, the graph provides an objective map. You could just use D3js, but for me the dot markup was faster on the command line than structuring json.
The idea is if you can formulate a conceptual, narrative ontology of an organization, you can create a grammar of things and relationships, and then you can plug data (contract counterparties) into that model and form a fairly complete map.
Another recent use case was enterprise vulnerability scan data over a very full /16 address block, allocated across multiple divisions under different management hierarchies with thousands of hosts. By linking the host ownership data to projects and an org chart with the types of vulnerabilities, I could demonstrate in a couple of slides what the highest impact patching strategy would be. Again, graphviz for sketching up the ontology, then Neo to do the lifting.
On a much simpler scale that was more graphviz/dot oriented, I did some work for a startup where I worked with the executive team who had acquired a codebase and talent, and created an ontology of their pipeline customers, their stated needs, implied product features, platform dependencies, our service interfaces, their code bases, and demonstrated the flow of how work on the code bases flowed through to impact revenue. This ultimately got represented as a Sankey diagram, but it was graphviz/dot I used to sketch up the initial ontology.
Graphs and infosec go very well together.
It's pretty notable that the coolest data viz is for discovery, whereas most managers just need to know whether the line has gone up or down. I'm thinking there may be a fundamentally different cognitive orientation to whether one is hunting for opportunity or managing a resource.
However, that was back in the late 90's. Now the internet has many examples of better looking layouts, but it is still disappointing that they don't look ... designed?
But integration and automation are great. Like GNUPlot, which I've used for decades because it is so easy to automate once you learn how to make plots look better than the default.
GStreamer uses DOT output by default. But any reasonably sized pipeline is almost impossible to read without excessive zooming. But it gets the job done.
https://github.com/enter-haken/schema/blob/master/schema.sh
It's a mixture of sed, awk and sql.
I'm not very proficient with graphing, so can't compare it to Graphviz, but a few examples I tried were relatively easy.
https://dot-to-ascii.ggerganov.com
It's useful to embed diagrams directly in source code for example.
Here is a sample [0]:
0.6
┌─────────────────────────┐
▼ │
┌───┐ 0.8 ┌───┐ 0.1 ┌───┐ 0.7 ┌───┐ 0.2 ┌───┐
│ d │ ─────▶ │ │ ◀───── │ e │ ─────▶ │ a │ ─────▶ │ b │
└───┘ │ c │ └───┘ └───┘ └───┘
│ │ 0.4 │
│ │ ◀──────────────────────────────────┘
└───┘
And a more complex example from a project of mine [1]: ┌─────────────┐
│ @tweet2doom │
│ │
│ ROOT │ ◀─────┐
└─────────────┘ │
▲ │
│ │
│ │
┌─────────────┐ ┌─────────────────┐
│ @player1 │ │ @player2 │
│ │ │ │
│ /play 50-u │ │ /play 30-l,50-f │
└─────────────┘ └─────────────────┘
▲ ▲
│ │
│ │
┌─────────────┐ ┌─────────────────┐
│ @tweet2doom │ │ @tweet2doom │
│ │ │ │
│ Video │ │ Video │
│ │ │ │
│ Start: 1 │ │ Start: 1 │
│ End:50 │ │ End:80 │
└─────────────┘ └─────────────────┘
▲
│
│
┌─────────────┐ ┌─────────────────┐
│ @tweet2doom │ │ │
│ │ │ @player2 │
│ Video │ │ │
│ │ │ /play 30-l │
│ Start: 80 │ │ │
│ End:110 │ ──▶ │ │
└─────────────┘ └─────────────────┘
[0] https://dot-to-ascii.ggerganov.com/?src_hash=476410d3I would kill to have this as like a Python library or something, any chance you’d open-source it?
Or is there a GitHub link already and I overlooked it?
Either way great job.
It's already open sourced on Github [0] (there is a link on bottom right of the page). Note that the page is simply a wrapper of Graph::Easy [1], so nothing technically interesting to see in the repo - it just passes the HTML input to a command-line tool and prints the result.
You can either use Graph::Easy directly on the command-line or you can use python to make an HTTP request to my page (example is shown in the README of [0]).
Here are a couple of my diagrams:
https://beepb00p.xyz/blog-graph.html -- graph of my blog pages with tags/connections between posts, generated with a DSL-ish python script https://github.com/karlicoss/beepb00p/blob/master/misc/index...
https://beepb00p.xyz/myinfra.html -- map of my personal data & infrastructure (discussed a year ago https://news.ycombinator.com/item?id=26269832 ). Also a similar DSL https://github.com/karlicoss/myinfra/blob/master/generate.py
The main downside for me is that sometimes it gets the positioning wrong, and you can see how it can be easily fixed, but it's hard to convince graphviz to actually do so. Basically I'd love a tool where I can do 10% of positioning manually and let the rest be constraint based like in graphviz.
Edit: dotty is what I was trying to remember.
Best example was for fabrication of a large subsea structure. Often, these critical assets have material traceability requirements on-par with space exploration, producing thousands of pages of certificates and documentation along the way.
I.e. you need to prove and assure the provenance of every piece of material sitting on the 1km down on the ocean floor back to the assay report of the soup in the ladle at the forge when it was formed, and everything in between; material certificates, design specs, third party testing, mother plate to material stock to final cut part identification, and all the third-party witnessing and assurance along the way. Easy for small parts but cumbersome for 60tn structures with thousands of parts and welds.
When the requirement is 100% QA/QC coverage, linking IDs to certificates and other documents in each part of the chain meant we could easily visualise things and look for rogue elements like child parts pertaining to be from two seperate mother plates, or destructive testing coverage for all materials.
Not how it's usually done but it was a useful tool for communicating to other not so technical stakeholders and helping some on the project sleep easier. :-)
For example, I had to parse DWARF debug info recently to scrape type information, something I dreaded doing. Instead of diving into the DWARF specs, I set out to adapt one of the pyelftools [1] examples to produce a .dot file and graphed it with dot, producing [2].
Now looking at the picture, it's nearly obvious how functions and structs and types are stored. The rest is trivia (How do I access this attribute? How do I iterate over a DIE's children?).
Here's a simple alias so that anything that writes to /tmp/tmp.dot can be viewed with a single command:
alias graph='dot -Tpng /tmp/tmp.dot -o /tmp/tmp.png && open /tmp/tmp.png'
[1] https://pypi.org/project/pyelftools/How Graphviz thinks the USA is laid out - https://news.ycombinator.com/item?id=25611053 - Jan 2021 (80 comments)
Create diagrams with code using Graphviz - https://news.ycombinator.com/item?id=23475225 - June 2020 (211 comments)
Graphviz – Graph Visualization Software - https://news.ycombinator.com/item?id=18797932 - Dec 2018 (3 comments)
Show HN: Project Management as Code with Graphviz - https://news.ycombinator.com/item?id=15950325 - Dec 2017 (64 comments)
A Quick Introduction to Graphviz - https://news.ycombinator.com/item?id=15324883 - Sept 2017 (44 comments)
A Technique for Drawing Directed Graphs (1993) [pdf] - https://news.ycombinator.com/item?id=14568180 - June 2017 (11 comments)
What happened to GraphViz? - https://news.ycombinator.com/item?id=11265188 - March 2016 (2 comments)
Ask HN: What is your favorite application of Graphviz? - https://news.ycombinator.com/item?id=790668 - Aug 2009 (8 comments)
There have been others, of course, but these seem to be the ones most focused on Graphviz itself.
1. No set-up, no accounts/registration.
2. Free, no ads.
3. Simple, pleasant UI with variable/keyword autocompletion and even multi-cursor find/replace!
4. Your graph is saved in the URL so sharing the link lets others play with a copy of your graph. I usually link-shorten the URLs since they're long:
https://bit.ly/edotor-example-1 https://bit.ly/edotor-example-2I've used it to visualise/debug everything from web scrapers in Python to directory scanning in Bash to proper graph algorithms in Java. Every time with no library installation or anything, just simple prints to a stream/file and perhaps a system() call at the end if I'm running it often enough.
.XML -> Python script -> .DOT -> Graphviz -> *.SVG
It's very useful and probably the best example of using DOT language, unfortunately.
Regarding Graphviz itself, I wonder why is there no special layout logic for planar graphs? They can be recognized and embedded on the plane in linear time without intersecting edges, so it would be very nice if some of the Graphviz tools actually did that.
A recent set of minimal graph coloring Graphviz visualizations of mine: https://gitlab.com/nsajko/example_optimally_colored_graphs
It's pretty handy to get a rough overview of a new project or to use in docs.
[0] https://github.com/django-extensions/django-extensions
[1] https://django-extensions.readthedocs.io/en/latest/graph_mod...
Edit: typo
I guess my use case is mainly swimlanes and places where nodes don't have to have the same rank.
My feeling is likely impacted as much from the fact that I've never met anyone that used it.
Exports to SVG, PNG, JPG
Odds are slim that its future owner will have the skills to work with such a resource. But I did leave a couple printouts on the boat before leaving it!
Graph technologies are tricky to build up -- you can go deep in many areas like this. Afaict, AT&T/Bell Research used to be the main developers of graphviz, and when that corporate lab thinned out, so did graph viz development. A similar story happened with gephi devs when they left their universities (was popular in social data), and cytoscape (popular in bio) is still largely at that fragile state.
We are historically big contributors to OSS at Graphistry, but saw the history of funding issues in this space, and how each tool reached a frustratingly low ceiling due to it. We decided to not start as open core, and instead made a free GPU tier + release individual pieces as OSS (Apache Arrow's JS tier, etc) + initially charge for GPU code, until we could build + grow more commercially targeted parts. Thankfully, we are getting to the point of enough sustainable revenue that we are pushing much more to our OSS libs now, and one of our OSS pushes for 2022 will be moving custom (GPU-accelerated) traditional layouts & AI layouts to the OSS lib pygraphistry :) A request we get often get for pygraphistry and our file uploader UI is dotfile (and graphml/gexf) conversion/ingest for then viewing+editing in our scalable interactive graph renderer, so we have been thinking of adding a pygraphistry example just to demonstrate the OSS flow!
That means it's not possible to flip graph/digraph to see how it looks with/without arrows, but naievely it doesn't seem that the difference makes any difference except making that difficult, if the connections must always be written left to right. (I've not used more advanced features, so it could be that it does more, or could be that it was built expecting people to know what they want first, which would all be fine)
Plotted the calls with graphviz and the questions were instantly trivial to solve. Great tool with many applications.
One thing I learned creating these graphs was that with some tinkering you have quite a lot of control on how the result will look, without littering the semantic in the source file with a lot of markup.
I choose graphviz because the dot notation can easily be checked into source control. It is also easy to create reproducible results using a makefile.
It is a great tool to have, it makes drawing complex diagrams manageable.
GraphVizio[1] "an addin for Microsoft Visio which allows you to layout Visio diagrams using the renowned Graphviz algorithms developed by AT&T research laboratories." - https://www.calvert.ch/graphvizio/
PSGraph[2] a PowerShell wrapper / DSL for GraphViz and dot language. - https://powershellexplained.com/2017-01-30-Powershell-PSGrap...
https://openprocessing.org/user/15252?view=sketches
Basically any sketch that has "3d" or "aspekt" in the title is on that track. But you can also see from the thumbnail many of them are 3d world related. The navigation of that site is pretty tricky...but you can find the source code by:
- Click on a sketch
- Click on the 'abacus' control menu to the right
- Click on the files tab in that
Source code in Java should be available there.
Please remember it requires a version of Processing in Java to work. I worked on this in 2011/2012 so you may need to figure out which Processing version that is.
Also, here's a youtube video of flying around one of my sketches:
https://www.youtube.com/watch?v=lNshrYO7Kf4
and here are some more videos:
Used it as a helper in my first project at Snowflake (role hierarchy layout) https://github.com/Snowflake-Labs/sfgrantreport and won a nice hacking award for it. Then the product saw it and folded what I made into the main UI.
Huge props to the graphviz for making it easy to draw my ideas from whiteboard into svg/pdf/png!
GraphDumpFile = /etc/tinc/network/graph.viz
will generate a real-time network graph using the Graphviz DOT language. It's a cool feature that I find quite useful.I also used it to make a small tool to generate the relationships of Elixir's modules: https://github.com/shiryel/graphme
I used it (with a various degree of success) to plot some diagrams for my personal usage.
I get that polymorphism can mess that up, so maybe just link callers to the base class in those cases.
PlantUML has the benefit that you can include it in your builds, e.g. parse any .plantuml files in `make doc` or in your CI.
But I like Mermaid more: its easier to read, less quirky and easily integratable in the markdown (or note-taking apps) without the need for extra build steps.
That's how I recently found that Mermaid can't handle graphs with no flow https://github.com/mermaid-js/mermaid/issues/1227
[1] https://wiki.dendron.so/notes/ba97866b-889f-4ac6-86e7-bb2d97...
- Viz.js (via Emscripten): https://github.com/mdaines/viz.js; Online demo: http://viz-js.com/
Edit: GraphQL Voyager has the memory increase hack https://github.com/APIs-guru/graphql-voyager/blob/54a178142a...
I've had good success with cytoscapejs as well, integrating with react so the graph will animate layout on transition. I use elk, which is apparently superior to sugiyama (the algorithm graphviz uses for force-directed dag layouts).
https://github.com/jamesshore/object_playground/blob/master/...
https://www.graphviz.org/pdf/dottyguide.pdf
Edit: I don't know of a really good visual tool as I've found text editing .dot files directly with a live visualization pretty nice. I use graphviz pretty regularly.
Because a lot of big companies use it (and they spend 0 €/$/... in the project), and it is broken from several years...
We do voluntarily act on CERT advisories.
But, true.