Databricks acquires serverless Postgres vendor bit.io
databricks.com
databricks.com
They've extended and their platform does a lot now.
I assume that’s Apache Spark, which is described as a “ unified analytics engine for large-scale data processing”
Still not clear for me what to use it for :-/
You can use batch processing, streaming, do machine learning and graph jobs. You usually use Scala, Java, Python or R to write your code. The code is executed in Scala, so it all gets converted to it. For example, in Python you'd use PySpark and that gets written down to its scala equivalent which is then executed.
I mainly work in Python, so I'm going to talk about some features there. But it support dataframes and exposes the data in Spark DataFrames. You build operations and those slowly build a DAG. It's not until you either execute, save or request to see the data that it actually starts executing the DAG after optimizing what it needs.
If you need something that spark doesn't support, you can use regular python, but because it won't get converted to spark, it'll run on only one node and be limited. So you have to rewrite your code optimizing for it.
You can process some data in memory, you can use disk, you can use databases. Either as source or targets.
A use case can be, load the raw data as it comes in, transform the data to your intermediary states, then write out different tables based on what they need to do.
---
It's a framework that has an engine to manage code running on clusters, a language to interact with the data, abstractions and optimizations of the code, ways to store the data, checkpoints for optimizations, and other things.
It's an ancient African word that means "I am because I can't install Apache Spark".
If you have the money, having a managed Spark instance with a bunch of added features can be a big win for some. There is a lot that goes into Spark maintenance.
Maybe there is no production db left from paying customers?
The naming is really confusing. When I brick my console it's broken. I'm not sure I want to brick my data :(
Call them platform engineers?
Hope nobody was using bit.io as a set and forget solution...
Which I thought was the entire point of the cloud hosted databases?
I'm not thinking about whatever legal technicalities could be debated by lawyers, but what real-world truth is.
This is crazy. 30 days to migrate? Hope nobody is taking a holiday in the next couple of weeks.
What does that say about their own products? What if you integrate their products and are locked to their platform without any easy migration options?
If they lose interest on one of their own services, you very well may have 1 month to move, and 2 months to have a chance at keeping your data.
Looks like I will have to go back to neon (they fixed the bug).
If anyone has other ideas, I'm all ears. Project is hosted on Cloudflare and they have D1 now, but Dittoed uses a little bit of PostGIS.
Would like to know more about the cost tradeoffs, also. Please elaborate.
Surprise!
Stinky.
Was easy to export my dbs from bit.io -- did so this morning.
Some Helm charts: https://github.com/neondatabase/helm-charts
It could potentially be one of their partners:
For the record though, they're not enough to run neon today[0] - this has been a "problem" since neon was announced here[1].
[0]: https://github.com/neondatabase/helm-charts/issues/35#issue-...
They have published a new piece which is how they vertically autoscale Postgres in Kubernetes: https://github.com/neondatabase/autoscaling
Autoscaling with live VM migrations is quite cool. Here is a blog post on it: https://neon.tech/blog/scaling-serverless-postgres
And yes, the code is open feel free to use it!
(In my opinion) when someone says here's the helm chart I assume running "helm install $THING" would give me a running version of $THING, so it's more so no one has the wrong expectations (like I would)
But I also needed to read their Ansible files to understand how they manage their infra better. Those are deleted now, but luckily you can just look at the history (commit that deleted it: https://github.com/neondatabase/neon/commit/0d3d022eb1fe4a42...)
Bitenporal support?
Smart anonymization (like Tonic.AI)?
Looking forward to hearing the announcements
I think there are a lot of different reasons why people may want to use a service like bit.io, but if you want a database with data in it to code against, run tests against, reproduce production related data-bugs, and run e2e tests against then check out https://www.snaplet.dev.
Databricks coined the "Delta lake" concept and are still (just about) leading the way, but Fabric has the potential from MS to take away that marketshare. Databricks need to improve their "serverless SQL" offering, and add a serious "data warehouse" component alongside the lake.
There are other fatal flaws to the Spark implementation in Synapse that I think carried over to Fabric. Worst one is the clunkiness/inability to run multiple notebooks concurrently on a cluster.
If anything, companies are drowning in a proliferation of siloed datastores and most are highly motivated to fix that situation; the exact opposite concern of "quickly spin up a new database".
Take "OpenSearch Serverless" for example: They claim "you only pay for the resources consumed by the workload", but even if you have an OpenSearch Serverless collection you don't use, you pay at least ~$690/month (and that's not even accounting for stored data)!
1. I don't have to think about or manage any servers
2. Usage is metered at a very fine-grained level (per X requests to the API/per GB of data/etc)
3. No fixed cost. You only pay for usage.
Was there a different meaning originally?
I don't remember those ever being called "serverless". Certainly "peer to peer" or "distributed" have a lot more traction.
Bit io is awesome. It just works. I mean so does elephant but bitio has more storage. I never got very far with my learning and never did tadvanced db concepts like cross apply though so it was just simple entities and tables but it worked just fine and the best part, no credit card required on file.
Fly sounds nice but I don't feel so good about having to give them my credit card number...
That said, mostly PG compatible data types, indexes and queries, horizontally scalable with pay for what you use, free and reserved tiers.
I don't understand what a company hopes to gain doing stuff like this as the (long term) incentives don't seem to align.
The executives of the firm being acquired usually don't come, unless they have some skillset the acquiring company needs. But they (hopefully) get a cash bonus for the successful acquisition.
Everything is negotiable of course.
Supabase gives you a full Postgres database, we position ourselves as a Firebase alternative because we offer a few other bells-and-whistles. The database is just postgres[0] and so it has more compatibility than bit.io offered[1]
[0] https://github.com/supabase/postgres
[1] bit.io compatability: https://docs.bit.io/docs/supported-sql
Foranyone looking, their documentation on how to connect, https://supabase.com/docs/guides/database/connecting-to-post...
But my understanding is they essentially got hired at Databricks? Maybe got a paycheck to do it?
Meanwhile they shuttered and abandoned the product and all customers.
Is it really the goal to make an mvp and plan to get noticed and acquired vs actually making a product, customers can migrate in a month or not we don't care?
The product they made was literally meant to be a reliable solution. 1 month for all customers to migrate away? Really? That's assuming they see the announcement today too, it would be so easy to miss the email/hacker news post.
I think I've seen this before