217 karma · joined December 28, 2011
Apart from the size of the document, there is no limit on the size of the array or the depth of nested data. We plan on substantially increasing the document size limit.
But supporting data type changes without rebuilding is not ideal. It will lead to data quality issues and complexity on the application. Integer -> String example is simple. But what about String -> integer, how are the consumers of data supposed to handle the situation where the field in some records has a string value and in some has an integer value? They will have to add type checking which complicates each of these consumers.
Then some of the downstream consumers such as data warehouses depend on strict data validations. We went through this problem at Uber- I blogged about it here https://www.uber.com/blog/dbevents-ingestion-framework/
Assuming you have declared your schema as shown here https://docs.tigrisdata.com/typescript/getting-started You can evolve it by updating your type definitions, deploy the new version of application. Once `createOrUpdateCollection` is called, it will update the schema.
--
The IDB idea sounds very cool, let me dig into it more.
However, if you are an application developer looking for a ready-made solution that you can plug-in as your application's backend, then FDB does require heavy lifting. For example, you would have to implement auth mechanism, query layer, schema management, and indexing. This is where Tigris comes into play.
You have a very interesting idea about backing IndexedDB APIs with FDB.
This is one of the most confusing aspects of the modern data infrastructure industry, why does every new system have to completely rebuild (not even reinvent!) the wheel? Vendors are spending so much time rebuilding existing solutions, they end up not solving the actual end users’ problems, although ostensibly that's why they decided to create a new data platform in the first place!
In this post we talk about our approach to building Tigris - the open source developer data platform. We talk about why we chose to build on top of FoundationDB, one of the most reliable distributed KV store with an amazing correctness story. We also go into detail about our experience using it.
There are essentially 3 major components: - FoundationDB (providing transactional data storage) - Sharded version of Typesense (providing in-memory indexes) - Query engine return in Go (stateless compute)
Each of these components can be scaled independently.
We will definitely be adding tutorials showing how Tigris eliminates complexity and provides a simple API experience.
You make a great point regarding the tutorial. Where would you want to see it? In tutorials, or would that be the first page you would want to land on as a developer?
`docker run -d -p 8081:8081 tigrisdata/tigris-local:latest` essentially gives you a local development version of Tigris to play around on your local machine as well as to allow you to build and test your application locally. We have a managed version of this in the cloud that essentially runs the entire platform on Kubernetes.
I am glad to hear that this is similar to what you have thought about building as well. As an application developer in my past life, I have seen so much pain building apps because eventually I end up spending more time on infrastructure and maintenance and less time on my application code. We have built Tigris to solve this by providing you with simple APIs to interface with data without dealing with a separate database system, a separate event storage system, a separate search system, etc.
We didn't go with graphQL because we want a more broad adoption and it seems like HTTP APIs are still the most widely used. At some point though I do envision us having graphQL support.
We are open source so would love your contribution regarding python client library.
We do want to speak to folks building new applications as Tigris takes away the infrastructure burden completely from the start. We want to bring the same API experience to "data" as you get when using Stripe API for financial transactions.
Migration is often times necessary. In my experience I have seen countless examples of migrations once an application becomes popular and the backend technology choices at that point either become brittle (hard to keep adding new app features), or do not scale. So I wouldn't categorize it as marginal maintenance cost.
The video, I agree could be improved. Because the pain is not just seen at the maintenance stage, it is felt at every step that you add a new technology to you stack.
Regarding your other comment about "people will go with whatever the default tool". It is actually the opposite, if you give people the choice they will stick to far fewer technologies :) This is why when you see someone building a top Postgres, as an example, initially they use it for the OLTP workloads, then they use it for the full-text search use cases (knowing fully well there is a better alternative because PG is not a search engine), or use it for analytics (even though there are other special-purpose analytical products), or use it for modeling event stream (even though again there is a special-purpose product available for event streaming). This is how application architectures organically grow, and the reason this happens is exactly because it is much easier (development time-wise and maintenance-wise) for me to continue to use X until it is unsustainable. And this is where we are coming from. We want to be that general purpose solution that you can continue to use to build up your application and it will grow/evolve with you.
We did prepare this use-case video, I am curious to hear your thoughts: https://youtu.be/G489hasvgio
Besides that, it is Apache2 license which means folks can deploy it to their k8s and run it.
At the core of it, it is a transactional document database that uses FoundationDB under the hood for data persistence. It automatically provides search capability on the data stored (similar to Elasticsearch) without you having to pipe data to Elasticsearch. Then there is event streaming capability that is also provided out-of-the box, so you can use simple publish/subscribe APIs to build event-driven applications without having to introduce Kafka.
A better comparison would be against MongoDB Atlas. Here are some of the ways it is differentiated from Atlas: - Open source that you can deploy anywhere, the only requirement being Kubernetes. Whether it is your cloud environment or on-prem, it doesn't matter. - Much simpler API interface as compared to MongoDB with native support for HTTP enabling it to be used from anywhere where HTTP is accessible - edge, mobile, etc. - Built with strong transactional consistency guarantee (strict serializability) - this is the benefit of using FoundationDB for data persistence. - MongoDB Atlas is based on 10 years old architecture and the industry has evolved quite a bit thanks to the cloud. Tigris has a micro-service based architecture with separately deployed components (compute, storage, indexing) that can each be scaled independently. - True serverless experience, which is of course possible through a modern architecture. - Automatic indexing, so you don't spend time optimizing queries and secondary indexes.
But we have built multiple features on top of it to enable it to be used as complete platform. Query processing layer that can do complex read queries similar to FoundationDB. Ability to do full-text searches similar to Elastisearch. Ability to use it for event streaming through simple and easy to use publish/subscribe APIs without the complexity of Kafka.
All of it is done in a cohesive manner, so for example, when you use it as your event streaming system, you get out of the box capability to query and search data without having to pipe the data somewhere else.
The reason it is more than a database is because it provides more than the structured data storage that is provided by databases.
At the core of Tigris is a transactional document database built on FoundationDB. Then Tigris automatically indexes the data so you can run search queries (think Elasticsearch) without having to pipe the data to Elasticsearch. Tigris also provides an integrated event streaming system so that you can build event-driven applications (think Kafka, Google PubSub, etc). The beauty of the event streaming system is that you can then also run search and read queries on the events data automatically.
We have explained the high level architecture here: https://docs.tigrisdata.com/overview/architecture
StoneDB uses the knowledge grid technology and a column-based storage engine.”
That doesn’t seem like a HTAP database but an analytical database. Transaction processing has completely different requirements which cannot be fulfilled by a columnar storage engine.
To be able to scale, the database would most likely be composed of several raft clusters which brings couple of other challenges: - You need a sharding layer so that the queries can be routed to the right cluster - You need an orchestrator to be able to manage these clusters
It gets trickier as you try to run transactions across multiple raft clusters. You end up needing to implement 2PC.
FoundationDB solves all of the above except for the query engine part (given it is a KV).
If you look at CRDB or TiKV design, under the hood they are also KV-stores with SQL layered on top.