Hacker News Reader

Comment by memhole | Hacker News Reader

Nice write up. I don’t think the comments about duckdb spilling to disk are correct. I believe if you create a temp or persistent db duckdb will spill to disk.

I might have missed it, but the integration of duckdb and the arrow library makes mixing and matching dataframes and sql syntax fairly seamless.

I’m convinced the simplicity of duckdb is worth a performance penalty compared to spark for most workloads. Ime, people struggle with fully utilizing spark.

View on HN

About spilling to disk, in DuckDB’s docs I see:

> Both persistent and in-memory databases use spilling to disk to facilitate larger-than-memory workloads (i.e., out-of-core-processing).

I don’t have personal experience with it though.

https://duckdb.org/docs/connect/overview.html

Miles Cole here… thx for the correction, another reader just notes this as well. I’ll get this corrected tomorrow and possibly retest after verifying I have spill set up. Thx!

Yeah I'd never go for spark again, all of its use cases are better handled with either DuckDB or Ray (or combination of both).

I thought Ray was a reinforcement learning platform? Can you elaborate on how it is a replacement for Spark?

Ray is a distributed computing framework that has a fast scheduler, 0 copy / serialization between tasks (shared memory) and stateful "actors", making it great for RL but it's more general than that.

I'd recommend checking out their architecture whitepaper: https://docs.google.com/document/d/1tBw9A4j62ruI5omIJbMxly-l...

Imagine spark without the JVM baggage and with no need to spill to disk / serialize between steps unless it's necessary.

Thanks. Can you clarify what you mean by your last point? One of the main advantages of spark over Hadoop is that it doesn't need to spill to disk/ serialise between steps, i.e. it is pitched as an in memory big data platform, so I'm a bit confused.

Ray has some really unfortunate nomenclature/branding in my opinion. Ray is, itself, a framework for distributed computing, on top of which they have built numerous applied platforms such as Ray Data, Ray Train, Ray Serve, etc. This includes your reinforcement learning platform, RayRLib.

I think they’ve diluted the “brand” a bit with this approach and would be better off sticking with “Ray” for the distributed computing and spinning up the others as something completely separate, but that’s just me.

Reply on news.ycombinator.com