jinmingjian

Location: China(UTC+8)

Remote: YES

Willing to relocate: YES

Technologies: Database, SQL, Rust, Java, JVM, Scala, WebAssembly, WASM, React, C/C++, Python, Javascript, Typescript, HTML5, CSS, TailwindCSS, Node, REST, Tauri, Linux, Kernel, IO_URING, BPF, LLVM, MLIR, LLM, LLAMA, ChatGPT, Finetuning, Vector Database, RAG, chatpdf, chatbi, chatexcel, BI, Analytics, Bigdata, Datalake, Data Warehouse, ClickHouse, PostgreSQL, Spark, Flink, Impala, Iceberg, Hudi, Kudu, Parquet, Arrow, SQL on hadoop, Redis, Distributed, AWS, GCP, Azure, SIMD, Lockfree, GPU, WebGPU, CUDA, ML, PyTorch, Triton, Compilation, JIT, VM, Docker, Cloud, MQTT, HTTP, TCP, IoT, Time series, Gateway, Ethereum, EVM, ZK

Résumé/CV: Send upon request

Email: jin.phd at gmail.com

14 years in work experience and 21 years in commercial software development, focused on engineering and science fields such as databases, compilers, deep learning, language design, high-performance computing, high-concurrency backend and network protocol and communication from data centers, to clouds and to decentralized internet.

MQTT and Kafka are different things. MQTT is not necessarily better or worse than Kafka, vice versa.

But, sadly, people are often caught in their own loops, when there are or can be created better options.

I have created a new, free, single binary data-service platform for IoT, JoinBase: https://joinbase.io/

This single binary data-service platform has proved:

1. Kafka is not more suitable for industry or higher performance than MQTT, even from the protocol level

With carefully crafting, JoinBase has saturated one PCIE 3.0 NVME sustained write bandwidth (25 million msg/s) in single modern node. This is, in fact, can not been done by the Java-based Kafka.

We have provided FREE full functionality community for testing: https://joinbase.io/products/

2. Use MQTT and Kafka together is unnecessary. This only makes your pipeline more complex, expensive, but much unstable and slower.

JoinBase can do arbitrary message preprocessing and auto-view(WIP). Streaming does not have to be owned through a separate monster.

3. High performance or ease of use, has nothing to do with the size of the software if the product can be properly engineered.

5MB Single binary JoinBase is enough to beat many monsters in the IoT/AIoT data pipleine: + sustained batch MQTT message write throughput: ~10x faster than Kafka and ~5x faster than that of one popular broker + basic SQL analytics: 3-4x faster than ClickHouse + HTTP interface concurrent queries: ~100x higher than ClickHouse + ... More could be seen in our 2022 summary blog: https://joinbase.io/blog/joinbase-2023/

There are historical reasons for all of these, of course. But it could be great that we break out of own mindset loops.

I am the author of JoinBase. It is interesting to see that one REST layer project on the top of PostgreSQL is up on the first page: https://news.ycombinator.com/item?id=34172205

JoinBase shares some basic ideas with PostgREST, but we provide a much simpler solution for the view of database: installation, configuration, authentication, schema creation and even performance are damn simple within JoinBase.

Shameless insertion:

We release our new HTTP interface with our free AIoT database - JoinBase:

HN discuss: https://news.ycombinator.com/item?id=34181591

or the blog link: https://joinbase.io/blog/http-interface/

Different to PostgREST, the HTTP interface of JoinBase is integrated in the database. So, no verbose setup, one binary rules them all!

Our HTTP interface is inspired from the ClickHouse, but we provide 100x message throughput than that of ClickHouse in the HTTP interface. JoinBase's HTTP interface is so fast that you can use it to provide unlimited production-level REST services without any worry.

If someone are interesting for JoinBase, just request the free distribution here: https://joinbase.io/request/

"Third parties to help" is a good point. Although under the USAL, you can make "third parties" to become users.

Thanks for the suggestion! The USAL is new, it may be updated to allow pluggable additional grants to allow more flexibility, for example, if the licencor as a business entity does not exist, then the licensed works could be changed to another license.

Copyleft does not solve the problem: how to build a positive business cycle. No company seriously contributes to the copyleft. That is why the Apache License is raising. And you can see that all changed licenses mentioned in the article are not copyleft. Copyleft is great for licencors, but most of companies in the world, does not want to share their customs. This is the businesses.

The Linux kernel is almost the only one in the world and cannot be copied.

YMMV.

But the Apache License is truly dead for startups. Most of "open source" startups are living on the VC investments. If a good business cycle cannot be built, the achievements of open source cannot be truly maintained. This is the author's usefulness.

USAL distilled:

* The license is clear that you can anything to the licensed sources except the (re)distribution.

* With USAL, the advantages of Apache License are inherited, while the disadvantages of Apache License are avoided. (for "open source" infra startups)

* BSL breaks the greatest feat of open source, USAL fixes it.

I am the developer of JoinBase and happy to answer any question about it.

The interests of JoinBase is that, in combination with kinds of scenarios, we explore the infinite possibilities of expanding database technologies.

The database is is dead, long live the database.

sorry, JoinBase now is just freeware, but not open-sourced. If you are interesting, you can request here: https://joinbase.io/request/.

Interesting reading, thanks for sharing. This is quick summary:

The articles compare the performance of different MQTT brokers implemented in different languages, including D, C, Erlang, Go, Java and C++. The basic conclusion should be D impl is the winner in the loadtest case, but C (Mosquitto) are the total winner for two case: loadtest + pingtest. C++ impl is not good. And the author said he "really don’t want to write C++ again unless I have to".

Basically, the conclusion of the article is nice. For MQTT brokers, the primary case sub-to-pub is just a layer-7 forwarding. This is an IO intensive scenario, if your MQTT parser is not written too badly.

So, for the impls done in system level languages, like C or D should archive the similar performance. Here, C++ impl should be an exception, although I haven't looked into the exact reason.

These blog articles, in fact, are quite old. The 100k message/second got with hundreds of connections is top ranked. However, the current Intel 12th Gen notebook can run at 250k mps with just 1 connection. I have tested (and read) several open-source MQTT clients and brokers. These MQTT brokers including impls done in C, Java and Erlang, can run at 1 million to 2.x million mps in a single modern Xeon socket but hard to go upward.

The truth here is that, the free lunch given by the efficient modern kernel TCP stack is just at 1-2M mps. If you want a faster broker, you need to make this kind layer-7 forwarding faster in the computational part, i.e. faster parsing and parallelizing. Traditional optimization tricks comes back again, like cache-friendly coding. The result is that we reached around 8 million mps in the same modern box even with message parsing, data routing and data dump all done in the first version of JoinBase(https://joinbase.io/).

The free lunch from the Moore's Law is truly over. We should be prepared for this.

Great. The OIDBS is using a built-in MQTT client. So, if you want o benchmark your server, just run the `import` command like as shown in the quick start video:

``` MODELS_ROOT=CURRENT_OIDBS_DIR import DIR_TO_DATA_SOURCE -n nyct_lite -d ```

here,

CURRENT_OIDBS_DIR is a hacky ENV var, for finding the models, should be removed in the future.

-n: is for the built-in model name, only the OIDBS repsect two models: nyct_lite and nyct_strip, you can get the two model sources from the article

-d: is for importing data only without injecting schemas. This is common if you are benching against a MQTT broker or server as you said. In fact, with this opt enabled, you can import any data source iff the data source is CSV or JSON. The benchmark logic is just to read and send the message line by line. That is, the files in the nyct_lite and nyct_strip directory is can be any CSV or JSON format. And the files'name is not important, but the model name still to be one of above two. Because the code checks the name.

(Ok, OIDBS may let the model name to be optional for such case, I will record this issue. thanks!)

If you have any problem, you can ask details in the community. I will help you as possible.

Thanks for all the 3 upvotes:) I am the developer of the OIDBS, and glad to answer any question here.

Great to see the this SBC benchmark. I have done one benchmark for several available SBCs from an edge data stack viewpoint (Edge Data Stack Capabilities Ranking: https://joinbase.io/ranking/, but the article is still in progress...so sorry for no content now)

What I have tested : $20 RockPi S(minimum squared Armv64 dev board), $0 unused IPTV box (S905L2 chip)(should be bought from secondhand market in ~ $7), #30 NanoPi Neo3(the minority of cheap SBC with DDR4 on), the Allwinner D1(the first batch avaiable Risc-v 64 board) and my own Xeon server (and more cloud based instances).

A possibly surprising fact is that, if the modern software (such as my database here) can take full advantage of the modern hardware, then the performance of even a small 40mm squared SBC board (such as RockPi S) can be compared to the large-scale-used traditional softwares running on a Xeon Platinum 8260 based bare-metal server(such as PostgreSQL).

Just another embedded SQL engine.

There are SQLite(OLTP), DuckDB(OLAP) and some engine-based project like mentioned Apache Arrow(https://arrow.apache.org/)(OLAP): Apache Arrow has many language implementations, some do not include the query engine(for example, Rust implementation, which depends on the DataFusion for more SQL-like analytics) in its own repo, but other do include(for example, C++).

There is a comprehensive benchmark by ClickHouse for OLAP but including kinds of embedding engines: https://benchmark.clickhouse.com/

The more interesting is that, in fact, we have not an embedded HTAP engine. One of my database products already implements 3/4 HTAP at the engine layer, but unfortunately it's still just a free software, not an open source implementation.

I am the founder of this database product. A fundamental curiosity in my heart is, after tens years of development, can the bottom layer of the common-users-accessible database still develop?

JoinBase, is my initial answer to this question.

1. For the first time, it proposes the evolution of the database from the user's point of view.

2. For the first time, it well supports three DB payload types in a single engine of a free industrial-grade database: TP single row transcation write, TP and AP read.

3. For the first time, it is so fast that for most businesses, it is no longer necessary to use a distributed database for their data needs.

[1] Benchmark: https://joinbase.io/benchmark/

OK. Seemly no too much interestings:)

As the first end-to-end IoT Database, it is interesting to see how we fit IoT domain data storage and analysis much better than existed open source projects.

I recommend one ClickHouse compatible OLAP database project in Rust: [TensorBase](https://github.com/tensorbase/tensorbase/) for anyone who likes working with AP-side DBs on Rust.

FYI, recent information and progresses for TensorBase:

1. TensorBase(TB, for short) is not an reimplementing or clone of ClickHouse(CH, for short). TensorBase just supports the ClickHouse wire protocol in its server side.

2. TB's in-Rust CH compatible server side is faster than that in-C++ of CH. TB enables *F4* in the critical writing path: Copy-Free, Lock-Free, Async-Free, Dyn-Free (no dynamic object dispatching).

The result of TB's architectural performance: the untuned write throughput of TB is ~ 2x faster than that of CH in the Rust driver bench, or ~70% faster by using CH own ```clickHouse-client``` command. Use [this parallel script](https://github.com/tensorbase/tools/blob/main/import_csv_to_...) to try it yourself!

3. Thanks to the Arrow-DataFusion, TensorBase has supported good parts of TPC-H. [Untuned TPC-H Q1 result here](https://github.com/tensorbase/benchmarks/blob/main/tpch.md).

4. In simple (no-groupby) aggregation, TensorBase is several times faster than ClickHouse. [Benchmark here](https://github.com/tensorbase/benchmarks/blob/main/quick.md).

5. For complex groupby aggregations, recently we help to boost the speed of the TB engine to the same level of ClickHouse(not released, but coming soon).

6. TB will soon supports MySQl wire protocol, distributed query, adaptive columnar storage optimization... Watch [issues here](https://github.com/tensorbase/tensorbase/issues)

Finally, it is really great to build an AP database in Rust. Welcome to join!

Disclaimer: I am the author of TensorBase.

sorry for late. I am the author of this project. Welcome to any question:)

It is often doubtful if one uses the word "fastest". You often see that one micro-bench lists ten products, then it says "look, I am running in the shortest time".

The problem is that, people often compare "apple to orange". Do you know how to correctly use ClickHouse(there are 20-30 engines in ClickHouse to use. Do you compare an in-memory engine to an disk-persistent-design Database?), Spark, Arrow... ? How can you guarantee to do a fair evaluation among ten or twelve products?

There is already a general purpose working-in-progress OLAP project written in Rust.

https://tensorbase.io/

1. TensorBase is highly hackable. If you know the Rust and C, then you can control all of the world. This is obvious not for Apache Arrow and DataFusion (on the top of Arrow).

2. TensorBase uses the whole-stage JIT optimization which is (in complex cases possibly hugely) faster than that done in Gandiva. Expression based computing kernel is far from provoding the top performance for OLAP like bigdata system.

3. TensorBase keeps some kinds of OLTP in mind (although in the early stage its still in OLAP). There is no truely OLTP or OLAP viewpoints in users. Users just want all their queries being fastest.

4. TensorBase is now APL v2 based. Enjoy to hack it yourself!

ps: One recent writting about TensorBase (and those compared with some query engine and project in Rust works included) could be seen in this presentation: https://tensorbase.io/2020/11/08/rustfest2020.html

Disclaimer: I am the author of TensorBase.

shameless insertion:

"raw csv processing at ~20GB/s" has been demonstrated in one my project as the tooling byproduct[1] based on Rayon(great Rust data parallelism library)[2] and wrapping of modified simdcsv[3] into one simple .rs.

just a little more:

1. simdcsv has severe bugs, so do not use it beyond demo.

2. the processing model of simdcsv still has rooms to good improvements(estimated 2x more, a.k.a. ~40GB+/s in memory in single modern socket should be achievable in some scenarios(no heavy string to complex language object conversions)).

[1] https://tensorbase.io/2020/08/04/hello-base.html#benchmark

[2] https://github.com/rayon-rs

[3] https://github.com/geofflangdale/simdcsv

sharing some thoughts here, in that I am recently developing a similar thing:

1. "Query 1.6B rows in milliseconds, live" is just like "sum 1.6B numbers from memory in ms".

In fact, if not full SQL functionalities supported, a naive SQL query is just some tight loop on top of arrays(as partitions for naive data parallelism) and multi-core processors.

So, this kind is just several-line benchmark(assumed to ignore the data preparing and threading wrapping) to see how much time the sum loop can finish.

In fact again, this is just a naive memory bandwidth bench code.

Let's count: now the 6-channel xeon-sp can provide ~120GB/s bandwidth. Then sum loop with 1.6B 4-byte ints without compression in such processors' memory could be finished about ~1.6*4/120 ~= 50ms.

Then, if you find that you get 200ms in xxx db, you in fact has wasted 75% time(150ms) in other things than your own brew a small c program for such toy analysis.

2. Some readers like to see comparisons to ClickHouse(referred as CH below).

The fact is that, CH is a little slow for such naive cases here(seen at web[1] been pointed by guys).

This is because CH is a real world product. All optimizations here are ten- year research and usage in database industry and all included in CH and much much more.

Can you hold such statement in the title when you enable reading from persistent disk? or when doing a high-cardinality aggregation in the query(image that low-cardinality aggregation is like as a tight loop + hash table in L2)?

[1] https://tech.marksblogg.com/benchmarks.html

U.2, a.k.a. SFF-8639[1], in fact, is the successor of SAS.(It also allows legacy SAS and SATA usage.) So, it has a stronger enterprise inheritance. And so, the enterprise clients can even use the same disk cases/cages without any problem when updating.

Compared to M.2 form, U.2 has two benefits:

1. support hot-swapping(SAS-age enterprise character)

2. better thermal performance/heat dissipation(for larger shape)

for data centers, the future will be the EDSFF family[2](goodness of U.2+goodness of M.2). M.2 can still stay for the consumer market.

[1] https://en.wikipedia.org/wiki/U.2

[2] https://www.anandtech.com/show/13218/ssd-form-factors-prolif...

corrected: 10GB -> 10Gbps, 100GB -> 100Gbps

Kudos!

VSC is an "redefined" and high hackable (largely thanks to the rich nodejs ecosystem) editing environment which you can do everything as you like. So, to pin it into an editor or an IDE makes less sense.

more technical imagination: the project lead Erich Gamma has long history works in the field of Eclipse and definitely knows the importance (and pain) of the code editing to a programmer. Then, I (and we) can expect VSC would go beyond the Eclipse and hunt the heart of coders from other IDE products some day:)

final shameless insert: recently I release an extension to provide Swift editing environment[1] to try to bring Go/Rust like experiment for Swift server side development. Now it works for both Linux and macOS.(I hope to investigate the possibility for windows 10 WSL after release 2.0 coming soon.)

[1] https://github.com/jinmingjian/sde

Service Unavailable...Hi, boy, your site may go down. But I like the name, although it has been used over and over again.

The memory safety of Rust is over consumed.

1. several small languages also introduce the type system to try to solve the memory safety problem. But all of them are less famous. Because there are many reasons that makes a language being accepted massively from other tons.

2. in many cases, it is not hard to do manual memory management. There are many great software done with manual memory management. Although I admit the quest to memory management is always wonderful for system. But go to the follow #3.

3. linear/affine type system[1] is not the panacea. The case of "used exactly once" is just a small case. Forcely to this pattern makes large boilerplates. And constraints and verifications to system can not be done many levels and aspects. Is this truly valuable to add all into type system?

4. memory safety of Rust comes with price, which have added many complexities and language burdens to itself. Who like to read the following function declaration?(just borrowed as example):

fn foo<'a, 'b>(x: &'a str, y: &'b str) -> &'a str

5. So, finally, the question arises: does the current form of the memory safety of Rust deserve as the hope of next industry language? I'm afraid...

[1] https://en.wikipedia.org/wiki/Substructural_type_system

I also stand on the kernel maintainer's side.

Kernel is not your AMD's kitchen-sink:) They are always fighting for any holes. The maintainers become the maintainers because the community think they has the qualified capabilities. The best way is to argue in technical side: for example, why you should have such complexities, how to control the complexities or something like. No technical response makes non-sense and low down your reputation in the community.

Ubuntu is crying^_^