majidazimi

Attend CppCon once and it totally changes your mind. Mind blowing number of pure educational talks. There is literally no other conference comparable in terms of technical talks. The rest should be renamed to AdCon.

Why the hell does everyone make a banking example when they want to talk about transactions? BANKS RUN ON EVENTUAL CONSISTENCY, PERIOD.

Another point worth mentioning is that, single transaction coordinator model is undervalued in community. Majority of systems have the following query pattern (sorted descending):

1. Single key read --> you don't need to contact coordinator

2. Single key write --> you don't need to contact coordinator. OCC with simple version number works here.

3. Multi-key asynchronous write (updating second, third keys using a pub/sub infrastructure) --> you don't need to contact coordinator in this case.

4. True multi-key transactions that should happen atomically --> need to contact coordinator

If a single coordinator can process 20-30K/s transactions of category 4, then total throughput of the system would be around 300K queries per second which is sufficient for many workloads.

Sorry for my bad English. Here is a better explanation:

In Bookkeeper, one producer can fence the current active producer from appending to log. If majority of Bookkeeper nodes respond with OK to a FENCE request, then the current producer can not append to the log. This helps implementing master-slave database replication quiet trivial. Master keeps appending to log, and slaves keep reading from log. In case master fails, one of the slaves sends a FENCE request to Bookkeeper nodes which reliably blocks current Mal-functioning master from appending to db log.

However in Kafka, this isn't a native operation. Any client can push to a topic. There is no easy way to safely do this.

I was looking forward to see how Twitter team actually handled this specific use case.

Both Pulsar and LogDevice solve the problem of far-away-consumers with kind of segmenting the log and storing each segment in separate machine to reduce disk seeks and page cache pollution (when c1 reads head of the log and c2 reads tail).

So technically they are better than Kafka. But as the article mentions, they just used SSDs to get around this issue in Kafka. By looking at article the only valid reason to switch to Kafka seems to be KStream.

I submitted a question in previous discussion link[0] but I didn't get an answer by author. But hopefully someone will answer here:

How did you solve producer fencing mechanism which was built into Bookkeeper with Kafka (Assume I need to limit only one client to be able to write to a topic)?

[0] https://news.ycombinator.com/item?id=18559284

How did you solve producer fencing mechanism which was built into Bookkeeper with Kafka (Without considering the write proxy)?

The problem with RabbitMQ is that, you can't rewind back. It's good for messaging. But if you need log-like abstraction you need Kafka which requires zookeeper to start with. I would appreciate single node message brokers (very much like RabbitMQ) that don't do any fault-tolerance and just focus on providing log abstraction.

LMDB's read performance crushes modern LSM databases. It's not an alternative to SQLite. It is well suited for read-heavy class of workloads.

Then any other database written in pure Go would be sufficient to keep the index.

I've read the WisKey paper.

1. Wasn't it simpler to just store the Key/Pointer(Offset) in RocksDB and put the values in the value log? Then the whole database just becomes a simple wrapper on top of RockDB(or whatever K/V database) with bunch of files as value log.

2. Why 16 bytes for value pointer? 8 bytes is sufficient.

3. If LSM size is around 5GB, screw LSM. I would rather use a simple immutable HashTable which gives you consistent snapshot out of the box for persisting to disk. Every 15 minutes launch a thread to serialize the index to disk.

Am I missing something or there is some more detail missing? I mean the reasoning to write a whole database from scratch isn't convincing.

Presto on top of Apache Ignite is basically an OLAP database (It just takes some effort to write the connector). It doesn't support transactional workloads though.

Welcome to eventual consistency.

> Sounds like resume-driven-development.

Sure. That's one of the responsibilities of a team lead: to help team members to work and gradually build up their CVs. I want my team to experiment new stuff and learn while working. But I also want to limit the risk boundary. The whole reason why younger people are leaving dinosaur companies is that no one in management layer lets juniors experiment and fail. End of the day junior devs also want to improve and develop.

Why not? I mean we don't use micro-services for what it is supposed to do. The application would perfectly be a monolith. We do it, just because it lets us experiment newer technologies in not-so-important services in a smaller scale without undergoing major re-work since when the change is big, everyone in management layer understandably becomes conservative.

Pulsar and DistributedLog solve two serious problems of Kafka:

1. Parallel recovery when a node fails

2. Limited read capacity of multiple consumers reading different parts of the log.

Log Device from Facebook also addresses these issues.

Depends on the scale. With few users, you'd better query database directly. With enough users (when it makes sense to introduce complexity), I would adopt Change Data Capture pattern. A small process reads/follows database logs (journal) and when a new post or comment is added, updates the cache asynchronously.

I speculate this is not true for China. ISPs in China have to deal with country-wide censorship system which has to be upgraded to support IPv6 traffic. This might be the reason why China has a low IPv6 traffic.

> Is there more use of IPv6 in the weekend?

Most probably yes. Since IPv6 is deployed more on consumers side rather than enterprises. So it's not surprising that over the weekend IPv6 traffic hits a peak.

I wasn't opposing you. I just tried to mention, how easy it is to loose to your rivals by just being born in a wrong country.

Hmmmmm, in case you are a middle eastern immigrant tackling a Ph.D. in the US, it takes around one year (including security check) to take a visa just to get into the US. :D Meanwhile your German rival has arrived 11 months before you and probably had applied for an internship by the time you arrived.

I can't stop myself comparing people. :D

Well, I like to think persistence matters more but getting into top 1% of everything requires more than determination.

1. Going from bad to good requires x amount of effort/persistence.

2. Going from good to best requires 1000x amount of effort.

When you've put 800x amount of effort, you'll realize this harsh bitter fact. You have already given up on your hobbies, family, social life, ... and you are still considered as "good+" not "best-". To bypass being-best-barrier you need more than persistence.

Things like IQ, family wealth, great coach, ... matter when you are approaching near top of the hill.

The problem I can see right now with scientific community is that, no one pays you to re-implement the same idea just to confirm that it is correct. Every professor wants shiny new innovation from her Ph.D student. No one wants you to experiment the currently published ideas.

I'm about to finish my master thesis. I implemented a couple of ideas collected from multiple papers in my thesis, and I can say all of those absolutely stunning results were just intelligently crafted experiments that are not applicable elsewhere.

Basically, my whole thesis is just to show that some paper was wrong and the idea is not applicable in another experiment.

Don't you need a central controller for seamless roaming?

Let's not forget that with slightly more money that Fastmail or Protonmail are asking for just one mailbox, MSFT is offering you a whole office suit plus 1TB of storage.

Your criticism regarding functionality might be true, but there is no reason for competitors to charge more.

I'm just a simple email user -- 50 emails per week -- and I don't keep them in my inbox. As soon as I'm done with them, I delete them. A simple 50MB inbox is sufficient for me. I just need an ad-less mail box. For me anything beyond 5$/year is expensive as hell.

Comparing Office 365, then yes it is very very costly. Such services should earn money at scale. The basic plan should cost around 5$/year.

The single reason I switched from Evernote to Onenote is that it is expensive. 5$/month for a note taking app? Seriously? With 6$/month, I can get a full office suit, an email with no ads, plus 1 TB of OneDrive. Evernote is a sort of app that should make money at scale by offering extremely cheap deals.

No it doens't, since a single partition is stored sequentially on one disk which limits the consumers to bandwidth of single disk (say c1 reads beginning of the partition and c2 end of the partition). But in the case of Pulsar c1 is most probably connected to a different node than c2.

True, but Kafka has two very annoying features built into it:

- There is no many-to-many log recovery whereas -- for example in Pulsar/DistributedLog -- logs are stored in small segments and distributed to multiple nodes.

- Read scalability. Since all the log is stored in one node (with some replicas) the readers are bound to single disk sequential read capacity. Again Pulsar stores logs in segments that are distributed among broker nodes which helps a lot when there are many readers.

External logging service is my favorite way of doing replication. It provides nice features. Specifically:

- Cross vendor replication which makes migration much easier.

- No dependency on vendor provided replication protocols.

- Ability to use in-app databases such RocksDB, SQLite, ...

- Upgrading DB nodes becomes way easier since they are totally separated from each other.

I didn't have problem with MacBook Pro 2017 keyboard regarding dust. My major problem is, it is noisy. Using it in library is a huge hassle. It totally disturbs everyone around me.