It would be nice if OP noted what caused the change in their opinion?
did datafusion gain some feature that they noted was missing in the previous article, or did something in their understanding click so they could overcome the previous issues?
did datafusion gain some feature that they noted was missing in the previous article, or did something in their understanding click so they could overcome the previous issues?
I tried using DataFusion as an in-memory tool, which was a mistake. If the graph fits in memory, Networkit, IGraph, etc. will almost always be faster. These tools cannot process anything bigger than the available memory.
So, I changed my approach. I wrote my own naive "disk checkpointer," offloading everything to disk and avoiding materialization. Although I was afraid that writing to and reading from the disk would be slow, it is surprisingly fast with DataFusion. The results are impressive: fast and out-of-core.
Sorry, this post is short and not very detailed. I did not expect it to be at the top of HN and receive so much attention.
https://github.com/Ladybug-Memory/icebug-graphframes-compari...
I thought of datafusion work and the rust implementation as a way to address the delta seen vs the Spark/JVM GraphFrames implementation (author is a major contributor).
Looking forward to more such innovations, which will benefit the ecosystem as a whole. Why would anyone want to use a pure python graph algorithm package unless they're dealing with toy graphs?
So then the question became pandas/polars/datafusion/duckdb/etc, must of which are rust/native. I'm curious myself why datafusion vs others, it's an interesting project :)