jkodumal

Not quite a Saab 900, but perhaps the Bollinger B-1? https://bollingermotors.com/bollinger-b1/

We've been using Sleuth at LaunchDarkly as a single pane of glass for all changes going out into production. The addition of DORA metrics is exciting-- as an engineering leader, Accelerate (https://itrevolution.com/accelerate-book/) is one of the few books I've read where the practices described meet reality. Having a tool track those metrics is a welcome addition.

One thing I'm curious about is Sleuth's approach to the "garbage in, garbage out" problem-- if I'm tracking (e.g.) MTTR, I've found that our teams aren't always perfect about tracking the start / end times of an incident. If the data's incorrect, can I modify it in manually?

JSON Patch (http://jsonpatch.com/) also uses it.

There's VSCO, Fluid, Roofstock, LaunchDarkly, and Uber coming soon. There's probably a few more great companies that I'm forgetting.

Fun fact-- it was started by one the founders of HipChat.

That's mostly right. We don't use ES as a primary datastore, though.

I mentioned this briefly in the article, but we thought about doing something with Lambda + API Gateway. But doing the math, 5k RPS pushed through API Gateway is about $1500 daily just to authenticate.

We've thought quite a bit about how to make this work as a service. The key to our architecture is that evaluating a feature flag for a user does not involve a remote call. We make that work by embedding a rule evaluation engine in our SDKs. When you request a flag, the user is compared against these rules (in memory) and served the appropriate variation.

We then use a streaming API to serve rule changes, so when you make a change to your dashboard, the new rules are streamed to all your backend servers within a few hundred milliseconds.

If you need even more resiliency, you can deploy a small service in your own infrastructure (https://github.com/launchdarkly/ld-relay) that allows you to persist these flag configurations in Redis.

We had a lot of operational experience with Mongo from our previous jobs at Atlassian, and started out storing almost everything in Mongo. As we scaled out, we migrated all of our high volume data into other stores-- analytics into DynamoDB, searchable data into ElasticSearch, etc.

The last remaining piece is our core data, and there's no strong push to move it out of Mongo.

I think it depends on the workload. Serving 4.6k static pages per second, cached on a CDN, is not too difficult. However, handling an analytics workload of 4.6K RPS is a little harder.

I was being cheeky :)

Events are stored in DynamoDB. We use mongo for "core" data like accounts, feature flags, etc, but none of the high throughput stuff touches MongoDB.

For temporary feature flags, we try to clean up as soon as possible. As part of the rollout plan for a new feature, we'll have defined what metrics we're trying to achieve, and once we're confident that we've met those marks, we clean up the flag.

We eagerly create "cleanup branches" and corresponding pull requests to remove a flag-- that front-loads the cleanup work and means we aren't context switching back to try and remember how to clean up a flag.

Also, we've found that in practice flags are usually independent-- testing permutations of flags shouldn't typically be necessary. People usually take an analogous shortcut with feature branches-- we test branches in isolation but don't usually re-test after merging.

Of course-- we have a dogfood instance of LaunchDarkly. We use LD as our plan permissioning system (the plans you see on our pricing page are managed by LD). We also use it for ops-- we migrated a key piece of our analytics infrastructure over to DynamoDB, for example, and controlled the rollout of that via a feature flag.

And of course, we'll launch new features behind a flag. I'll admit that we've had two or three occasions where we had to hit the kill switch after a deploy, so I'm rather glad we had a LaunchDarkly for our LaunchDarkly.

[edit] here's a shot of our dogfood instance: https://twitter.com/jkodumal/status/717786744750477312

Our goal is to build a developer-focused platform that gives teams the ability to adopt feature flags as a core part of their dev cycle-- including pretty much all of the use cases described in Fowler's article (ops, permissioning, etc.). Many of our customers are not using us for A/B testing at all.

re: conditionals, DI: our SDKs focus on the base technique (flags based on conditionals), but it's possible to wrap that core with higher-level approaches like DI, etc. We're considering going down that path, but where it makes sense to do so, and not have to re-implement higher-level wrappers for every framework out there. So, e.g.-- in Ruby, I can see us providing higher-level APIs specific to Rails.

Dependent feature flags are also an interesting problem-- we don't have a solution to that yet, but we do spend a lot of time thinking about feature flags, and I hope we'll have something on that front soon :)

We do have some features to help manage the lifecycle of feature flags. For example, we can determine when a flag has been "flipped on" for everyone, and notify you that it's time to remove it. We can also determine that a flag has been removed from your codebase, and prompt you to remove it from LD (http://blog.launchdarkly.com/launched-flag-status-indicators...).

We have more coming, including an ability to mark "permanent" flags that should never be removed.

The product is built API-first-- everything in our UI is driven via our own REST API. Docs are here: http://apidocs.launchdarkly.com/

We've been using DeployHub at LaunchDarkly for a bit now, and it's a great tool to have in our arsenal. The thing I like about it is that it focuses on doing one thing well. Too many tools in this space try to control the deploy process too-- we've already got ansible and other tooling that works great there, but providing visibility + metrics on our deploys is super valuable.

Co-founder of LaunchDarkly here-- we're dropping our startup package to $9 / month in the next week. Come check us out.

Thanks for the shout-out! One of the cool things about LaunchDarkly is that you can use it out-of-the-box without needing to provision any additional infrastructure.

By default, the SDKs use server-sent events to push feature flags to an in-memory store. If you need persistence + strong consistency, you can back the SDK with Redis as well.

Disclaimer: I'm one of the co-founders of LaunchDarkly.

We do have multiple feature flags in place at once. Our dogfood server (we use LaunchDarkly on LaunchDarkly) has about 30 feature flags going at any given time, and we're a relatively small team (still one standup).

For the most part, flags are independent. When I first started using DVCS (back in the Bitkeeper days) we all thought that merge conflicts from doing feature branching would be a headache, but this didn't end up being the case. I've found the same with feature flags. While it's possible for flags to interact poorly, or to introduce "dependent" or nested flags, it hasn't been a problem for us in practice.

This seems to scale, too-- I haven't heard this to be an issue in practice at larger orgs (Etsy, Netflix, etc.) that do feature flag-driven development.

I haven't heard of module flag-driven development-- would love to hear more details.

We manage the cleanup issue by defining a "removal branch" that cleans up the feature flag at the same time the feature flag is introduced. This sticks around as a dormant pull request during the lifetime of the flag. It's not much overhead when this is done early. Cleanup does become painful when the code's not fresh in your mind.

Note that we do feature branches as well as feature flags-- trunk vs. branch based development is orthogonal to using feature flags.

CTO of LaunchDarkly here. We kind of embrace it a little bit-- locavore booleans as a service.

But the real power behind the idea is not the ifdef piece, it's the idea of dynamic, context-aware configuration. You've got a user-friendly tool that can control the code paths being executed in your application without restarts or redeploys.