tylertreat

Agreed with setting the standard arbitrarily high. It depends on what problem you're trying to solve—just like deciding when to use any other tool. Context matters.

Where I work, we've been trying to build some super-fast systems and things like the GC and even calling interface methods matter. We open sourced some data structures that show just how optimized we're trying to get (https://github.com/Workiva/go-datastructures), but in hindsight, Go might not have been the right tool for the problem.

I don't know if I completely agree with this. To write truly performance-critical Go, you end up throwing away many of the language's qualities. Channels are slow, defers add overhead, interfaces add overhead (e.g. I2T), etc.

Don't get me wrong, I like Go, but in my (and others I work with) experience, it's not the right choice for performance-critical systems.

Fair criticism :) The reason I mostly focused on various Bloom filters was because I actually encountered a problem at work recently that required them to deduplicate messages in a distributed messaging system. Mostly was journaling my research.

It would be fun to look at stochastic ways to approximate k-NN and other sketches.

How do you maintain data consistency at scale?

Not yet. The blog post linked in the readme there provides some more background on the motivation behind the project.

I'm hoping to do some in-depth analysis of several brokers at some point, but I want to get the benchmark instrumentation right first.

Really nice write up. I'm curious if you guys have done any extensive throughput and latency benchmarking? I saw the note saying "ballpark figures."

FWIW, I've been working on a framework for empirically testing queue performance for scaled-up, distributed deployments (https://github.com/tylertreat/Flotilla). Haven't gotten around to adding support for AWS services yet, but would be interesting to see how they compare.

I think pfctl is the suggested replacement for ipfw, which was deprecated in Mavericks. I wrote the Comcast tool someone linked in this thread. Haven't had a chance to dive through the pfctl man pages to add support for Yosemite yet unfortunately.

Yeah, for the record, the benchmarks I've run aren't at all scientific. Should also be looking at percentiles. That said, surge looks promising so nice work, Jian :)

My dream would be to build a framework which orchestrates testing across distributed senders/receivers. That would give you a much more accurate representation of performance and reliability instead of a single machine.

Thanks for sharing, that's a really interesting read. Also would have been very helpful while writing my own implementation, although maybe it's better I didn't beforehand. It's more fun to work on a problem without any preconceptions.

I believe you're confusing goroutines with channels. Goroutines are lightweight threads of execution. Channels are the pipes that connect goroutines.

Got it, that makes more sense.

Thanks, I'm not much of a C programmer, so I appreciate the feedback. I'm not sure what you're referring to with the mutexes and pcond not being on the struct because they are. Can you clarify? You are right with the disposal though, I need to clean that stuff up :)

Persistence was disabled. Nano and ZMQ are in their own group and aren't even attempted to be compared to other groups.

Yes, you're correct. "Benchmark" is a very unfortunate misnomer.

For anyone interested, I did a write up on Iris a few weeks ago, including how it stacks up to other golang messaging libraries. http://www.bravenewgeek.com/iris-decentralized-cloud-messagi...