I gotta be honest, I don't understand how this is possible. Like aren't systems like this chaotic? Shouldn't this be impossible to predict?
1,664 karma · joined October 11, 2010
aronchick (at) expanso (dot) io for work
hnchat.com:CPlr1G0P41k5zhCjiKQC
The problem is that the data is spread everywhere - no choice about that. So with that in mind, how do you query that data? Today, the idea is that you HAVE to put it into a central location. With tools like Bacalhau[1] and DuckDB [2], you no longer have to - a single query can be sharded amongst all your data - EFFECTIVELY giving you a lot of what you want from a data lake.
It's not a replacement, but if you can do a few of these items WITHOUT moving the data, you will be able to see really significant cost and time savings.
Disclosure: I co-founded Bacalhau
Yes, we're deeply inspired by all the work in distributed computing which came before us - Hadoop, Spark, Kubernetes, etc... there are so many. We're just trying to focus on the case that feels unmet by these centralized solutions (not a replacement, a compliment).
You can read more about our architecture here[1], but to get to your answer, we have two forms of consistency. First, when creating your network, you create a series of "requestor" nodes - as many as you'd like, in as many places as you feel you need to have redundancy. Unlike something like Kubernetes, where the state is managed by etcd (or other store), we manage that internally. That's not saying etcd is bad! But by managing this ourselves, it allows us to be smart about distributing job state between many requestor nodes around the network - spinning up new ones, and having them join the consistency group is just one command.
Further, when a node takes on a job, the state (such as it is) gets transferred to that node as the source of truth. This is a lot like Kubernetes but with much better support for "coast mode" - if that node is disconnected, but still in the timeout window, the system is perfectly fine with it, and when it reconnects, everything resolves. (In the centralized world, things are blown away and rescheduled quickly)
We looked at things like CRDTs and are currently doing a lot of thinking about more resilient ways to handle partitions and consistency - if you'd like to jump over to our slack[2]/repo[3], we'd love to talk more!
[1] https://docs.bacalhau.org/getting-started/architecture
Oh, one other thing - we do support IPFS [4] natively but it's optional; we also support S3 [5], local disk, etc.
[1] https://docs.bacalhau.org/examples
[2] https://blog.bacalhau.org/p/distributed-compute-platform-bac... <- look for pluggable executors (it's in beta)
[3] https://docs.bacalhau.org/examples/workload-onboarding/rust-...
[4] https://docs.bacalhau.org/running-node/storage-providers/#ip...
[5] https://docs.bacalhau.org/references/job-specification/input...
The idea is that this provides a more distributed friendly way to do the above. So imagine you’d like to run DuckDB queries over flat files spread in 7 different regions/zones/etc
Kubernetes is great, but requires using a central control plane.
Swarm is also great but is more focused on job execution, rather than targeting data, results, etc
We love both ecosystems (our cofounders and team are made up of very early contributors to both), but we think there’s room for a compliment of something that is more resilient to poor network conditions and allows for data first scheduling and in-place execution of any arbitrary job (containerized or not).
Does that help?
Disclosure: I co-founded Bacalhau
As always, we'd love to hear more if you have any questions/comments/concerns.
aronchick (at) expanso (dot) io
Disclosure: I co-founded Bacalhau
We (the bacalhau.org[0] project) are interested in helping with this - one of our philosophies has been that part of the problem is with that first step. By first moving to a lake of some kind, you end up giving up lots of optionality. Even basic things like aggregation, filtering, windowing, etc now need to be in the "locked-in" tool, which is exactly the wrong first step to take.
SHOW HN: We have a solution that uses DuckDB to do some of this initial work[1] which can save you 70%+ or more on total data throughput. Further, it allows you to do interesting things like eventing, multi-homing observability data, etc.
I'd be very interested to hear any/all thoughts!
[0] https://github.com/bacalhau-project/bacalhau
[1] https://blog.bacalhau.org/p/bacalhau-x-duckdb-deploying-appl...
Disclosure: I co-founded the Bacalhau project.
Doing distributed queries across many machines / buckets is super annoying today. We're hoping to make it easier to use things like Segment Anything / DuckDB in a distributed way with Bacalhau[1]. If you have any questions, would love to answer them!
dave aronchick (at) expanso (dot) io
[1] bacalhau.org
disclosure: I co-founded Bacalhau.
I'm David Aronchick - co-founder of Bacalhau[1] - and we are specifically building an open source platform to help with this. Basically, we allow you to do compute over data, making it much easier to schedule jobs (container, WASM, arbitrary binary) to where the data is being stored, even over irregular/unreliable networks. We were able to show 90%+ savings in things like log aggregations. Happy to answer any questions if you'd like!
David Aronchick aronchick@expanso.io
The editorialization came from the original author.
The data that you may be looking for is here: https://twitter.com/I_Am_NickBloom/status/170814596350283380...
The data that you may be looking for is here: https://twitter.com/I_Am_NickBloom/status/170814596350283380...
Disclosure: I co-founded Bacalhau
Unlocks a big speed up in ML workloads! Up next, WASM :)
If you’d like us to support something else, just let us know!
This feature gets us something I wanted for YEARS on Kubernetes (:) - I love k8s! Promise!). Externalizing the job queue lets you archive, achieve eventual consistency, and do analysis of your jobs. Now supported!
Super excited about this release. Please feel free to let me know if you have any questions!
Just a few of the features we released include: - Multi-architecture support - Intel, Apple Silicon (M1/M2), ARMv6 & ARMv7, AMD64 - Running 10k+ jobs simultaneously - Simplified private cluster setup - DAG execution through Project Amplify - Insulated Jobs & Federated Learning
And lots more! Please let us know if you have any questions :)
Disclosure: I am co-founder of Bacalhau
Compute and data are being created and run everywhere, we need platforms that understand how to use it and get insights without (or before) moving it.
Our contribution: https://github.com/bacalhau-project/bacalhau (think Kubernetes but built for the distributed world).
Disclosure: I co-founded the Bacalhau Project
Waterlily.ai: https://github.com/bacalhau-project/Waterlily Bacalhau: https://github.com/bacalhau-project/bacalhau FVM: https://github.com/filecoin-project/ref-fvm
We'd love your thoughts on our project - if you have any questions, please ask away!
Disclosure: I work on Bacalhau[1]
We're proud to announce Amplify - a simple way to augment your jobs with additional information, such as metadata generation, security analysis, and so o. Let us know if you have any questions!
Disclosure: I lead the Bacalhau project.