500 million requests per second. I think that's awesome for _intra thread communication_. I have a sharded bank simulation which naively sends money between accounts and it gets 700 million transactions a second which is the aggregate of 12 threads all transacting local to a thread, so 500 million across threads seems good (I split money up rather than accounts per thread). I see that some web frameworks can handle up to 600,000 requests per second on techempower benchmarks. So we need to combine the best of each!
Especially the part about one writer. I have a journal entry called "Golden concurrency" inspired by someone's comment on HN (I really should have linked it!) talking about writing to a database from 3 processes and they decided to split the database into 3 databases rather than suffer contention from concurrency control.
"Golden concurrency" is my term or desire that degradation caused by multiple producers/multiple consumers in different configurations could be optimised for and could be a scenario that is handled without performance degradation. I suspect you need a different design if you have 1 producer:1 consumer producers to consumers, or 1 producer:N consumers or N producers:1 consumer or N producers:N consumers.
I enjoyed the following benchmarks with Kafka and RedPanda which made me think of this too.
https://jack-vanlightly.com/blog/2023/5/15/kafka-vs-redpanda...