rockwotj

Oh if you care about durability on Kafka vs Redpanda, see https://www.redpanda.com/blog/why-fsync-is-needed-for-data-s..., acks=all does not fsync (by default before acknowledging the write), so it's still not safe. We use raft for the data path, a proven replication protocol (not the custom ISR protocol) and fsync by default for safety (although if you're good with relaxed durability like in Kafka you can enable that too: https://www.redpanda.com/blog/write-caching-performance-benc...).

As for Redpanda vs Kafka in multi AZ setups and latency, the big win in Redpanda is tail latencies are kept low (we have a variety of techniques to do this). Here's some numbers here: https://www.redpanda.com/blog/kafka-kraft-vs-redpanda-perfor...

Multi AZ latency is mostly single digit millisecond (ref: https://www.bitsand.cloud/posts/cross-az-latencies/) and the JVM can easily take just as long during GC, which can drive up those tail latencies.

Yeah be data driven and optimize the hot data path, the other code paths optimize for simplicity (and the majority of a system is the latter)

When the founder started it was designed to be two things:

* easy to use * more efficient and lower latency than the big resources needed for Kafka

The efficiency really matters at scale and low latency yes but the simplicity of deployment and use is also a huge win.

Plenty of people choose Redpanda because it’s the easiest getting started experience for kafka. There is a single binary for the full broker, where I have never seen Apache Kafka as easy to setup. It’s got a great UI as well.

The quickstart to getting everything running locally all documented here: https://docs.redpanda.com/current/get-started/quick-start/#d...

Disclaimer: I work at Redpanda

Redpanda is much more lean and scales much better for low latency use cases. It does a bunch of kernel bypass and zero copy mechanisms to deliver low latency. Being in C++ means it can fit into much smaller footprints than Apache Kafka for a similar workload

SQLites alternative in rust is a thing though

https://github.com/tursodatabase/limbo

The kafka protocol is a distributed write ahead log. If you want a job queue you need to build something on top of that, it’s a pretty low level primative.

ANN search is usually memory bandwidth limited from a search speed prospective, so it doesn’t surprise me that the CPU has a few extra cycles to decompress without losing much latency

Are you surfing the web on a toaster? I only run into network bottlenecks or pages that suck because it loads tons of ads, which isn’t a javascript issue really. Yeah its not Wasm fast, and JS has some terrible stuff from a language perspective, but I don’t blame a slow web on Javascript. Certainly not for everyday browsing

Honestly for Java I really like Bazel. You should give it a shot. I have a project with a self contained jvm and jars from maven central. Its more explicit than the other options but way less magical IMO.

It’s in the works from my understanding. I helped build the Redpanda Connect one and it’s quite easy to use

BTW I think most the ecosystem has settled on https://github.com/twmb/franz-go being the best and highest performing kafka client for Golang (purego)

https://github.com/artie-labs/reader is another I know of

lots of Go based CDC stuff going on these days. Redpanda Connect (formerly benthos) recently added support for Postgres CDC [1] and MySQL is coming soon too [2].

1: https://github.com/redpanda-data/connect/pull/2917

2: https://github.com/redpanda-data/connect/pull/3014

Yeah understood that the internal meta stuff would be too tied up with internal infra to be OSS, it’s the same with a lot of Google’s tooling here.

I do wish there was a startup here. There is sourcegraph, which has ok code search (github has come a long way, but without indexing and understanding the build you can’t do it justice). There are also cool code review startups like Graphite, but they don’t work together. I remember how powerful it was to review a change then go use an xref to see how a function is used that is untouched in a code review, so does not show in the diff, which requires checking out the changes locally in OSS land and context switching to leave comments.

It tries to bypass the kernel as much as possible. It runs in userspace.

I mean memory is completely allocated up front and fragmentation is something for application developers to deal with.

RE ScyllaDB, there will absolutely be no fork and it’s very unlikely they have ever had a meaningful contribution to Scylla OSS (which is not changing, just going to bit rot and the enterprise version which was closed source was moved to source available). The reasoning being the bar is very high for contributions. It’s C++ 20/23 without virtual memory and a userland cooperative thread per core scheduler (this is the underlying seastar framework). The skills needed to do anything meaningful here is very high and I don’t see the motivation behind it as I would expect people having feature requests would be customers, have an easier time extending Cassandra or the feature would be available in the Enterprise version.

Is there any UIs for this available openly? Or for glass? I am a former Googler and I know how awesome this kind of tooling is and it’s so hard to achieve with OSS. I would love open source code search. This seems very close but there is no UI layer (and it seems like meta uses this for code review and for IDEs) but a basic UI would be a good start

Swap out FAISS with usearch, you get all the awesome SIMD acceleration (via dynamic dispatch), optional compression. Not affiliated but really cool tech.

https://github.com/unum-cloud/usearch

I have never heard someone calling Github code reviews anything close to optimal. Here is where it falls short for me - do you not hit these?

- Main view is not the code, nor can I approve from that view, so it’s always half a dozen clicks to review even something trivial. - The main conversation view is very broken for me. Conversations always get lost or forgotten. - Any long conversation gets hidden behind another button and trying to load or find something at that point is very hard - There is a concept of resolving threads but no way to see if there are any unresolved threads

I am probably missing some but these are the largest issues for me

I was just going to say this, but you said it better than I would have!

Everything is just assembly!

I mean lambda urls have some kind of masking going om, but the docs still state it might be possible to determine the account ID:

> Lambda generates the <url-id> portion of the endpoint based on a number of factors, including your AWS account ID. Because this process is deterministic, it may be possible for anyone to retrieve your account ID from the <url-id>.

I always found it interesting that bucket names are a global resource and not per account (so there would be an account ID prefix in the URL)

Probably because there is no RAII semantics in JS and they don’t want to allow forgetting releasing the lock. Although the promise workaround is explicitly opting into this behavior

I believe that streaming changes from SQLite is what https://sqlsync.dev/ is

I worked for a short time on SearchGPT, and I can tell you OpenAI does respect robots.txt , at least when I was there and does now. They are also careful to shard per domain and only crawl each domain at a small rate (~1 qps) as to not ddos the site. OpenAI also uses User Agent strings to identify itself: https://platform.openai.com/docs/bots

They have dedicated user agents for search crawling, when a user directly asks about a site and for training data.

US folks, opt out here:

https://www.linkedin.com/mypreferences/m/settings/data-for-a...

> Firebase Realtime can be more efficient, as it can tell if a permission check has passed statically. I am not sure if Firestore works this way.

Firestore's rules are also able to prove before the query runs if the query will only return data that the user has access to according to the rules. That's a pretty important property that "rules aren't filters" because it prevents bad actors from DDOSing your system. My former colleague wrote about this: https://medium.com/firebase-developers/what-does-it-mean-tha...