agallego

You can use redpanda with interrupts just fine (no fan noise). You don’t need to busy poll for low resource envs. Same for sys alloc

Hmm. Strange. DM me details. Haven’t heard of anything like that.

What we found with RPCN (redpanda connect)/old benthos is that most systems are very slow and only cpu intensive things require manual CPU instruction optimizations like the snowflake connector we wrote (https://docs.redpanda.com/redpanda-connect/components/output...). The bulk of it is just about completeness. Go feels like the Perl of the 2020s. Cool little libs for just about everything.

It is not FUD. It is deterministic. Reproducible on your laptop. Out of all the banks I work with only a handful of use cases use rf=5. Defaults matter, because most people do not change them.

coo cool right on.

Redpanda cloud doesn’t limit tput. Most ppl get a bigger discount at high volumes. We have customers in 10s of GB/s. Confluent has those volumes too.

incorrect. the intend is to have it be a project that is thriving, see the last 2 additional partnerships that landed as apach2 connectors: https://redpanda.com/blog/redpanda-connect w/ peerdb, and ockam.

we trippled the team. added 3 meaningful connectors for CDC and zero-trust as well multi-lang SDK and kept 99% of the connectors available for ppl to make money on... as well as the core engine remaining MIT. This is about them not wanting to depend on redpanda products which is ok, but the whole thing is hard to believe from a company that has no open source products. it's more like "hey, i don't like it."

you may have not read the blog post i wrote. the engine remains MIT because we had customers that had embedded this in their app and it made sense to keep that. it is 100% about not having to call it "redpanda x" https://redpanda.com/blog/redpanda-connect

at the end of the day, there is plenty of ppl that are making money on this that is not us and that's cool too. we just need to retain the brand of the code we maintain. that's really the thing that matters.

let's call it what it is. warp never reached out. they do not want to have the name "redpanda" in their UI. that's all. They can* make money on 223 out of 225 connectors. More over the engine* remains MIT.

Hey jannesan this is untrue. We pay for ZoomInfo which gives us emails on search engines results. No one is manually filtering on your GitHub. But it should have a link to unsubscribe. Let me know if it doesn’t work and happy to remove. We use a mailing list provider which manages the unsub for us.

Oh that’s cool. How do plugins/add ons work with clickhouse

500MB for such a complete product is tiny! The largest CDN in the world ships 10GB+ binaries. 1G is common for large code bases if you link things statically. The bloat tends to come from transitive dependencies, most direct code is small in size

Curious as to how the code base evolved after forking from clickhouse

that's a littlebit of a stretch. when you say "no shortage" - outside of redpanda what product exists that actually compete in all deployment modes?

it's a misconception that redpanda is simply a better kafka. the way to think about it is that is a new storage engine, from scratch, that speaks the kafka protocol. similar to all of the pgsql companies in a different space, i.e.: big table pgsql support is not a better postgres, fundamentally different tech. you can read the src and design here: https://github.com/redpanda-data/redpanda. or an electric car is not the same as a combustion engine, but only similar in that they are cars that take you from point a to point b.

def. that should be the case, we have iot companies pushing us on a single pthread a a few megs of ram.

this seems truthy but isn't in practice. a lot of work, my perf optimization team @ redpanda does (yes we have a full team chasing tail latencies) is spent on CPU optimization, debouncing, amortizing costs, metadata lookups, hash tables, profilers, etc. so there is a lot of additional work after the IO layer which a decent async eventing thing can get you to get good perf.

That benchmark was comparing apples to oranges. Redpanda fsynced to disk and kafka saved to memory with deferred writes. Here is a response https://redpanda.com/blog/why-fsync-is-needed-for-data-safet...

this is a burner account created at the time this post was up.

what makes you think that we haven't tested this? seastar's io engine is defaulted to io-uring... there are about 10 things here to comment on. on optimized kernels we disable block coalescing at the kernel level, second we tell the kernel to use fifo, etc. these low hanging fruit was already something we've done for a very very long time.

exactly. you improve latency and throughput at the same time. is kinda cool. by delaying the reponses just a small bit you get huge benefits

it tends to be true. we do things at the application tier to minimize these things.

for example, assume you have to write A, B, C ... in syscalls it would be

write( A ), flush() write( B ), flush() write( C ), flush()

so if you add a debounce of say 4ms you get

Write( A ), Write( B ), Write( C ), Flush()

and saved 2 flush()-es

This is common with protocol aware storage applications.

the blog posts mentions that it is for any protocol that is non bft

That’s right. Raft uses sync writes which is what redpanda uses.

Complexity and industrial level reference impls are crucial

Right. But that line of thinking gets you to a place where one is like “how does anything work” haha.

In general this was a response to Confluent attempting to dismiss fsync() as a neat trick rather than an actual safety problem and why when we benchmarked we showcase the numbers we did.

the opposite should be true tho. opt-in for unsafe. you are the minority if you read the docs, let's be real :) most ppl never read the full docs. of the ppl i chat w/ is more like 5%

Following up - https://redpanda.com/blog/why-fsync-is-needed-for-data-safet...

Try this on your laptop see global data loss - hint: multi-az is not enough

Regardless of the replication mechanism you must fsync() your data to prevent global data loss in non-Byzantine protocols.

totally. we built a new team focused on the dev experience of k8s alone. 90seconds to prod (on a working eks cluster) with TLS, external certs, etc. That's the benchmark we're trying to hit :)

no, because it is built into the raft protocol itself. with Acks=-1 we only acknowledge to the producer once data has

1. writen to majority 2. majority has done an fsync()

i can see in the future giving people opt-out options here tho.

repeating things does not make them true. I read the post. You can only control some failures, but happy for us to write our thoughts in blog form.