Building a Distributed Data Warehouse Without Data Lakes
blog.bacalhau.org
blog.bacalhau.org
what makes it better than docker swarm or kubernetes or nomad or any other container orchestration tool ?
The idea is that this provides a more distributed friendly way to do the above. So imagine you’d like to run DuckDB queries over flat files spread in 7 different regions/zones/etc
Kubernetes is great, but requires using a central control plane.
Swarm is also great but is more focused on job execution, rather than targeting data, results, etc
We love both ecosystems (our cofounders and team are made up of very early contributors to both), but we think there’s room for a compliment of something that is more resilient to poor network conditions and allows for data first scheduling and in-place execution of any arbitrary job (containerized or not).
Does that help?
Disclosure: I co-founded Bacalhau
Kind of sounds a bit closer to something like Spark, where you want to distribute the compute over several machines, and with a few more executions modes/capabilities, is that correct?
> Kubernetes is great, but requires using a central control plane…something that is more resilient to poor network conditions and allows for data first scheduling
This sounds very interesting. I’m curious about your consistency model here, how are you preventing cluster members from getting out of sync, losing work, etc? CRDT’s? Make jobs idempotent by controlling side effects?
Also, love the name, the pun on compute-over-data is excellent.
Yes, we're deeply inspired by all the work in distributed computing which came before us - Hadoop, Spark, Kubernetes, etc... there are so many. We're just trying to focus on the case that feels unmet by these centralized solutions (not a replacement, a compliment).
You can read more about our architecture here[1], but to get to your answer, we have two forms of consistency. First, when creating your network, you create a series of "requestor" nodes - as many as you'd like, in as many places as you feel you need to have redundancy. Unlike something like Kubernetes, where the state is managed by etcd (or other store), we manage that internally. That's not saying etcd is bad! But by managing this ourselves, it allows us to be smart about distributing job state between many requestor nodes around the network - spinning up new ones, and having them join the consistency group is just one command.
Further, when a node takes on a job, the state (such as it is) gets transferred to that node as the source of truth. This is a lot like Kubernetes but with much better support for "coast mode" - if that node is disconnected, but still in the timeout window, the system is perfectly fine with it, and when it reconnects, everything resolves. (In the centralized world, things are blown away and rescheduled quickly)
We looked at things like CRDTs and are currently doing a lot of thinking about more resilient ways to handle partitions and consistency - if you'd like to jump over to our slack[2]/repo[3], we'd love to talk more!
[1] https://docs.bacalhau.org/getting-started/architecture
It’s a distributed OLAP database of sorts. More like a snowflake/ ClickHouse/ Trino (nee Presto).
I haven't been following the project, but I remember finding the original demo to be very impressive - I don't remember the details off-hand, but IPFS was somehow involved. Seems like a lot of work has taken place since then. Docs are here [2]
[0] https://www.youtube.com/@bacalhau3295
Oh, one other thing - we do support IPFS [4] natively but it's optional; we also support S3 [5], local disk, etc.
[1] https://docs.bacalhau.org/examples
[2] https://blog.bacalhau.org/p/distributed-compute-platform-bac... <- look for pluggable executors (it's in beta)
[3] https://docs.bacalhau.org/examples/workload-onboarding/rust-...
[4] https://docs.bacalhau.org/running-node/storage-providers/#ip...
[5] https://docs.bacalhau.org/references/job-specification/input...
As always, we'd love to hear more if you have any questions/comments/concerns.
aronchick (at) expanso (dot) io
Disclosure: I co-founded Bacalhau
The problem is that the data is spread everywhere - no choice about that. So with that in mind, how do you query that data? Today, the idea is that you HAVE to put it into a central location. With tools like Bacalhau[1] and DuckDB [2], you no longer have to - a single query can be sharded amongst all your data - EFFECTIVELY giving you a lot of what you want from a data lake.
It's not a replacement, but if you can do a few of these items WITHOUT moving the data, you will be able to see really significant cost and time savings.
Disclosure: I co-founded Bacalhau