The Bacalhau Vision – A Distributed Compute over Data Platform
blog.bacalhau.org
blog.bacalhau.org
First off, congratulations on the launch of Bacalhau and your continued work in the Kubernetes, Kubeflow, and SAME project ecosystems! The vision of bringing compute to data is an exciting one, and it has the potential to address many challenges related to data privacy, latency, and efficiency in distributed systems.
I had the opportunity to explore the Bacalhau project on GitHub and the documentation on your website. I appreciate the effort put into creating a user-friendly experience, especially with the integration of runnable examples in Colab. It's clear that the project aims to lower the barrier to entry for developers looking to leverage distributed compute over data platforms.
A few questions and suggestions come to mind:
1. Can you provide more information about the performance implications of running compute over data? How does it compare to traditional approaches, and are there any benchmarks or case studies available to better understand the benefits?
2. Security is a significant concern when dealing with sensitive data. Can you elaborate on the measures taken by Bacalhau to ensure the privacy and security of data during processing?
3. In terms of scalability, what kind of workloads is Bacalhau best suited for? Are there specific industries or use cases where you see this platform making the most significant impact?
Lastly, I would encourage the Bacalhau team to engage with the developer community by hosting webinars, workshops, or hackathons. This will not only promote the project but also provide valuable feedback from users to help refine the platform further.
Congratulations once again, and I look forward to seeing Bacalhau's continued development and its impact on the distributed computing landscape!
1) Yes these are HIGHLY scenario specific, but doing benchmarking and detailing is a great point, and we want to do it. 2) We have lots of thoughts about security - but it's all about just use your existing security profile FOR NOW. E.g. if you want to run data on something in private, and the machine would have access, then you can reuse that. You can also use things like env variable, endpoints, etc to provide credentials. We have LOTS of things planned, but I'd love to hear what you'd like first!
For running on the PUBLIC web, it is not for private workloads right now. You could encrypt everything you'd like, but even then, you'll still possibly run on an untrusted node, so not recommended. Instead, choose a trusted provider (e.g. GCP, AWS, etc) to provide the compute, and provision your nodes, and you're good.
3) We have tons of thoughts there too - basically anywhere you have more than 10 devices with/generating data and 100GB, you're PROBABLY running into a compute over data problem. This is many fold! Security, web serving, financial services, IoT, Edge, Fog, multi-cloud, etc etc. We are in the midst of narrowing down to some brain dead ones and providing scenarios and templates - expect lots of this to land in the next few weeks/months.
If you have some that you'd like us to work on, please let us know!
THANK YOU SO MUCH FOR YOUR FEEDBACK. You can also find/chat with the whole team in the #bacalhau channel in Slack [1]
Disclosure: I co-founded the Bacalhau project.
[1] filecoin.io/slack
We've recently opened up a public-hosted cluster (all runnable from colab in our docs [4]) and would love your feedback - you can see our vision at the attached blog post. Thanks!
Disclosure: I co-founded Bacalhau :)
[1] https://kubeflow.org/ [2] https://sameproject.ml/ [3] https://github.com/bacalhau-project/bacalhau/ [4] https://docs.bacalhau.org/