Comment by motohagiography | Hacker News Reader

Comment by motohagiography | Hacker News Reader

Parent Full thread

motohagiography

Graphviz is an amazing tool, it has just suffered from being cloistered to technologists whose jobs aren't to solve the kinds of problems it is specifically adapted for. I use graphs in security and privacy quite a bit, and even built the tech for a hopeful security platform using a graph back end. They yield the fastest path through complex problems, and I use them to do in a couple of hours what typically would take client staff months.

They're really compelling but oddly unpopular.

Imo, they're useful for one-off pattern discovery, and they're most valuable for finding single or a few exceptions and outliers in normalized data, and you need to be in an environment where there are asymmetric returns on finding those. Surveillance seems to be their default use case, with open ended scientific research a close second. This comes up with graph based recommendation engines, which are essentially a surveillance/marketing product based on preferences. Most businesses aren't based on discovery of anything other than customers for a transaction they already have.

These aren't problems engineers typically solve, which are more about scaling and optimizing, they're more marketing and sales problems, where you're looking for exceptions and opportunities. (security and privacy are the complementary antithesis of these) A graph based product (imo) is ideal for product marketing analysts optimizing for customer preferences and discovery.

From a product perspective, graphs are analogous to ML, where you'd use a clustering algorithm on loosely structured data to yield categories, comparisons, and implied relationships, whereas a graph yields the same thing over the structured normalized data you can feed into it once you have imagined an ontology to fit it.

That sounds really interesting! Can you give a few more details how you use Graphviz? How does it give such a great advantage "to do in a couple of hours what typically would take client staff months."

Did I understand it correctly that you use it to discover patterns? Are these patterns discovered by just using the layout engines? Arent' other tools, e.g. networkx in Python or cytoscape in javascript easier to use interactively in a REPL? What is the typical workflow (maybe plot, find interesting pattern, change query/data in a loop)?

I'm really interested in how Graphviz can be so great. I am currently working with the other mentioned tools for visualization purposes.

Really simply, I typically use Neo4j, but if I have flat homegenous data, I just use Graphviz because the dot markup format lends itself to parsing easily in awk command lines. The times I have used networkx was when I needed a graph abstraction layer to reason about another graph query, so networkx wasn't used as a persistent graph store, but more of an intermediate data structure for orchestrating multiple service and api calls, like a low rent graphql. I'm a crap developer, but the graphs were what i needed to piece the logic together coherently.

One example of clients taking months is mapping counterparties to agreements. Let's say you have inherited a division that has file share full of contracts and you want to understand the line of business. You get the counterparties out of the contracts and find all the paths for obligations between entities within the division and their counterparties. The graphviz/dot layout gives you a map of all those parties in a single slide and shows clusters, instead of a 3 lb. document with a paragraph for each of them that would have cost a massive amount of consulting time, or interviewing several people to get their narrative understanding of how the business worked, the graph provides an objective map. You could just use D3js, but for me the dot markup was faster on the command line than structuring json.

The idea is if you can formulate a conceptual, narrative ontology of an organization, you can create a grammar of things and relationships, and then you can plug data (contract counterparties) into that model and form a fairly complete map.

Another recent use case was enterprise vulnerability scan data over a very full /16 address block, allocated across multiple divisions under different management hierarchies with thousands of hosts. By linking the host ownership data to projects and an org chart with the types of vulnerabilities, I could demonstrate in a couple of slides what the highest impact patching strategy would be. Again, graphviz for sketching up the ontology, then Neo to do the lifting.

On a much simpler scale that was more graphviz/dot oriented, I did some work for a startup where I worked with the executive team who had acquired a codebase and talent, and created an ontology of their pipeline customers, their stated needs, implied product features, platform dependencies, our service interfaces, their code bases, and demonstrated the flow of how work on the code bases flowed through to impact revenue. This ultimately got represented as a Sankey diagram, but it was graphviz/dot I used to sketch up the initial ontology.

Have you tried gephi? It's not exactly an alternative to graphviz (eg. you can't cluster nodes) but it handles much larger graphs and has a bit more flexibility in layout. It has plugins for both dot and neo4j input.

Graphs and infosec go very well together.

Wow that looks cool. It reminds me of what Orange.app was for regular data viz, this Gephi is for graphs.

It's pretty notable that the coolest data viz is for discovery, whereas most managers just need to know whether the line has gone up or down. I'm thinking there may be a fundamentally different cognitive orientation to whether one is hunting for opportunity or managing a resource.

Gephi is fantastic, unfortunately it it hasn't seen a new release since 0.9.2 in 2017.

I found the learning curve to be quite steep with few examples of graphs that looked well-designed. By default, large graphs look like crap because engineers designed it, not designers (it takes both!).

However, that was back in the late 90's. Now the internet has many examples of better looking layouts, but it is still disappointing that they don't look ... designed?

But integration and automation are great. Like GNUPlot, which I've used for decades because it is so easy to automate once you learn how to make plots look better than the default.

GStreamer uses DOT output by default. But any reasonably sized pipeline is almost impossible to read without excessive zooming. But it gets the job done.

Reply on news.ycombinator.com