Show HN: Ggraph – A graph visualization library for big messy data | Hacker News Reader

Show HN: Ggraph – A graph visualization library for big messy data

gransk.com

76 pointspcbje32 comments

It's hard to get a sense of how it'll look with 'big messy data', when it's sample dataset is so tiny, at 14 nodes and 21 edges. Is there a way to see how it'll look with a ton of data?

Still not big, but a bit bigger: https://gransk.com/ggraph-bigger.html (Nodes: 611 Edges: 2499)

Edit: It's an excerpt of the Enron dataset.

To merge nodes based on search results and min-edit distance is currently not included, but probably will be quite soon.

Thanks, that really helps.

It's getting noticeably laggy for me at 611 nodes, but below you say it shouldn't be a problem to do 5k. Have you actually tested that kind of data?

Yes. Animating node placement has a significant performance penalty, so in larger graphs you would either hide edges or everything during the first couple of seconds.

Like you said, still not big and its starting to get laggy. Do you have an example that actually is big and not laggy?

Are you referring to node placement or selection/merging? Node placement is done with D3 force directed layout, so I don't think I'll be able to do much there. And again, animation makes it more soggy.

Thanks, I had the same question and you answered it nicely — I will stick with Vis.js (http://visjs.org/network_examples.html) for now. It seems to work better for larger graphs and allows me to build a pretty good ui for exploring the graph, too.

Yeah, when I think of big, I automatically think trillions. But, I'm jaded. lol

I was hoping for something that maybe looked like a set of hierarchical clusters. Where I could drill down level by level.

Perhaps related, I came across this graph library comparison page recently. You can try out the different libraries with different graphs and sizes:

https://anvaka.github.io/graph-drawing-libraries/#/all

How big can the graph get before taking a noticeable performance hit? A better example should be shown if advertising "Big ... Data"

It's built on top D3 and has the same limits (5k nodes 20k edges should not be a big problem). The "big" here is the ability to combine nodes, for two reasons:

1 Reduce the number of nodes and edges, thus increasing capacity

2 Combine nodes that should be seen together (e.g. alternative spellings and typos), to better deal with the variation-aspect of big data

I see your point, but the goal here was to illustrate the functionality.

I don't have the time to try it right now, but can it visualize 20M-100M nodes?

What would the use case be? Humans can't parse that visually. At a number way lower than 20M, nodes and edges should be culled.

At some point you run out of pixels on your screen to visualize it all at once. My suggestion in your case is to not use a graph at the beginning, but rather grouping nodes based on metrics. Like degree (group all contacts of a node that only has a single contact), communities, or value (e.g. IP's in a subnet).

To illustrate the functionality, the point of the original poster is a valid one. The interactions for managing 5k nodes are not necessarily the same as those for managing 10 nodes - not just in terms of performance, but also in how to make sense of all the information shown; so showing a realistic use case is required to assess the possibilities of the tool.

On a side note, the "right-click to select nodes" doesn't work well in Firefox, with its "always show context menu" option. At the "up" event, the context menu shows and the selection is erased.

In general, using right click on web applications is a terrible idea, since the standard function in browsers is to show a context menu for the clicked element. IMHO a lightweight selection mode or quasimode would work better.

Noted. I think you're right, I just hate checkboxes and radio buttons.

Maybe drag with a modifier key pressed? I guess that doesn't help for tablets though.

Try this one on for size.

https://gransk.com/ggraph-bigger.html

Congratulations! This seems to be a great tool to effectively display large graphs.

Maybe you can submit it to Neo4j team to be showcased in their "Graph Visualization for Neo4j" section: https://neo4j.com/developer/guide-data-visualization/

We used https://www.graphistry.com to visualize hundreds of thousands of edges for https://www.yesgraph.com/twinmaps/

The minimal eng required to get good performance was wonderful

For the yesgraph link I had to disable ad-blocking and uBlock Origin and still only got a static image with a Twitter advertisement pasted over the image.

The graphistry link is similarly useless just showing a few sentences on a few pages ending in a "request demo" button.

Hi Ivan, always cool to see what Graphistry users are doing!

Phil, sad to hear you weren't able to see your Twin Graph. Many of us use ad blockers and this is the first report we've gotten like yours, so we'll dig in. Meanwhile, you may be able to try a direct link to my own YesGraph TwinMap: https://labs.graphistry.com/graph/graph.html?dataset=lmeyero... . (Note: best on laptops, and we recently relaunched with Falcor/React, so currently porting all our page load optimizations.)

For more information about graphistry, we have users piloting the three below layers of our stack. Because we can load 10-100X more data at the visual tier than other systems here (so 100K-1M+ things), people have been exploring connections across events/entities for some fascinating reasons:

* Investigation & Response -- Connect to systems like Splunk and get rich, scalable visual graph views and easy workflow automation. Ex: build an investigation template that takes an indicator of compromise and runs queries that connect it to various users, devices, alerts, etc. Or, "here are our ssh trails and anomalies around them."

* Exploration: Data scientists and data analysts will explore connections in their events or samples, e.g., for week-over-week model tuning, security research & forensics, & even now loan analysis. They'll load in a bunch of events or samples where each may have a lot of attributes (IPs, times, amounts, ...), and then they can see correlations. Ex: most false positives are from events with 3 particular combinations of characteristics, or an outage involved 4 distinct phases of behavior and entities.

* Developers: folks building internal apps for scenarios like the above.

For the latter two use cases, a good place to get started is our API: https://github.com/graphistry/pygraphistry . Feel free to contact us at info@ if this may solve a problem for you. (And.. we're hiring! Help us build web-based visual tools with GPUs acceleration to solve real data problems!)

What are typical use cases for this?

To better understand complex relations in the data you are analyzing, e.g. friendship in social networks.

Do you know of any good sources for example data like that? Would be fun to explore.

Stanford has a good collection of network datasets: https://snap.stanford.edu/data/

The lasso seems to "give up" if too many nodes are selected - stops updating the selection although it keeps updating the green shaded area. Perhaps it should only give up on the text labels which are probably the expensive part.

I really like the grouping features and the labels work pretty well too (though it's slow with large groups). Thanks for sharing your work!

Thank you! I know, there are plenty of things yet to be done to make this thing run smoothly.

"Hold right click and drag" doesn't work on laptops with trackpads.

Reply on news.ycombinator.com