Disclaimer: I work for TiDB.
The incomplete list of TiDB adopters: https://pingcap.com/docs/v3.0/adopters/ and some of their adoption stories: https://pingcap.com/success-stories/
If you read mandarin, here are more: https://pingcap.com/cases-cn/
1,732 karma · joined October 8, 2016
The incomplete list of TiDB adopters: https://pingcap.com/docs/v3.0/adopters/ and some of their adoption stories: https://pingcap.com/success-stories/
If you read mandarin, here are more: https://pingcap.com/cases-cn/
https://www.pingcap.com/blog/chaos-practice-in-tidb/
Regarding the Fault Injection tools we are using:
- Kernel Fault Injection, the Fault Injection Framework included in Linux kernel, you can use to implement simple fault injections to test device drivers.
- SystemTap, a scripting language and tool diagnose of a performance or functional problem.
- Fail, gofail for go and fail-rs for Rust
- Namazu: a programmable fuzzy scheduler to test a distributed system.
We also built our own Automatic Chaos platform, Schrodinger, to automate all these tests to improve both efficiency and coverage
tl;dr: Chaos Engineering
Some links: https://cdn2.hubspot.net/hubfs/4466002/Using%20Chaos%20Engin...
https://www.pingcap.com/blog/chaos-practice-in-tidb/
Regarding the Fault Injection tools we are using:
- Kernel Fault Injection, the Fault Injection Framework included in Linux kernel, you can use to implement simple fault injections to test device drivers.
- SystemTap, a scripting language and tool diagnose of a performance or functional problem.
- Fail, gofail for go and fail-rs for Rust
- Namazu: a programmable fuzzy scheduler to test a distributed system.
We also built our own Automatic Chaos platform, Schrodinger, to automate all these tests to improve both efficiency and coverage
Keyword: SLUB: Unable to allocate memory on node -1
Bug #2: Network device reference count leak
Keyword: kernel:unregister_netdevice: waiting for eth0 to become free. Usage count = 1
Here are some links that might be helpful:
- the typical use cases: https://pingcap.com/blog/architecture-and-use-cases-of-a-clo...
- the key differences between MySQL and TiDB
https://pingcap.com/blog/5-key-differences-between-mysql-and...
- the current case studies: https://pingcap.com/success-stories/
https://openpowerfoundation.org/wp-content/uploads/2018/04/J...
The conclusion is as follows:
- Serialization may cause significant bottlenecks in big data frameworks
- Prevents effective deployment of accelerators in some cases
- Apache Arrow can help to alleviate bottlenecks
- We created an FPGA interface generation framework for Arrow
- Fletcher works with SNAP
- wraps almost only syscalls and uses only Go standard types;
- exposes standard interfaces; plays well with standard packages like io, bufio, etc.
- does not handle memory management (allocating/destructing slice). It's up to user to decide whether/how to reuse buffers.
Looks like it is effective for small data sizes and pure read-oriented workloads with heavy key distribution. Not sure whether this could work well for writes and uniform random distribution.
The past solution of the separate operational database and data warehouse poses great challenges for real-time analytics because it needs either data pipeline or the ETL process which could be the bottleneck of being "real-time", not to mention the waste of time, efforts and human resources maintaining multiple data warehouses. It was impossible for real-time analysis because, in the past, you would need a data pipeline, or message queue with the equivalent throughputs with your OLTP database, which I believe does not exist.
However, whether to adopt this hybrid solution depends on your specific usage scenario. For cases where users want to do real-time analysis in their data warehouse upon the same data table as in their OLTP database, TiDB is your choice.
- hybrid transactional and analytical processing (HTAP), coined by Gartner, - hybrid operational and analytical workloads (HOAP), by 451 Research - Translytical, by Forrester
If that's the solution you want to explore, TiDB (https://github.com/pingcap/tidb), the open source distributed scalable HTAP database, might be able to help you. ETL is no longer necessary with TiDB’s hybrid OLTP/OLAP architecture.
Here is a use case about how it helps the largest B2C fresh produce online marketplace in China to acquire real-time intelligence:
https://www.datanami.com/2018/02/22/hybrid-database-capturin...
Here is a tutorial about how you can try TiDB/TiSpark on your own laptop using Docker Compose: https://www.pingcap.com/blog/how_to_spin_up_an_htap_database...
Disclaimer: I work for TiDB.
TiDB has been widely adopted by many users (https://github.com/pingcap/docs/blob/master/adopters.md) in production because it support the best features of both RDBMS and NoSQL. It is quickly evolving and iterating based on users’ requirements which are prioritized and listed on the Roadmap.
At least, TiDB, CRDB, RethinkDB are open source :)