Hacker News Reader

> Ultimately, it turned out to be one of our dependencies turning on a feature by slowly increasing the probability it was used.

Wow. I feel like this dependency should be named and shamed.

Big company internal dependency. So nothing for the public to care about.

What company. I've seen this being done (and my team does it a lot at Amazon) but curious to know if others are doing it at build time too.

If done in a company with a monorepo I'd be especially interested in hearing more

> If done in a company with a monorepo I'd be especially interested in hearing more

Are there any big companies left which haven't adopted a monorepo?

AWS. We probably have the worst build systems :(

This is astonishing. The build (and deploy) systems are, by a considerable margin, the things I miss most about having left Amazon (CDO, not AWS, but still). What do you dislike about them?

I'm in a similar boat. I worked in both CDO and AWS. I miss Brazil and Isengard every day

I do concede that isengard is pretty cool. I am looking forward to Peru as it will make dealing with dependencies a lot easier.

I had to spend days trying to debug why a build that was working fine broke completely and NPMPM was unable to find the dependencies for the lambda.

I disliked how needlessly convoluted the pipelines are, and how some person pushing on accident to mainline can break everything.

So many things seem to be done the hard way.

> I had to spend days trying to debug why a build that was working fine broke completely

Sure, but I bet that it was helpful to have the `bats` tool so that you could replicate the build locally, right? As compared with other build systems where (so far as I can see - though I may be wrong) you basically have to push a debugging change for replication.

> I disliked how needlessly convoluted the pipelines are, and how some person pushing on accident to mainline can break everything.

This is true of any CI/CD system, though? In any system, if there's no push-protection set up so that you can only merge into main(/line) once a change has been reviewed (and run your tests at the point of review so that you know the merge won't break anything), you have only yourself to blame for breakage.

> So many things seem to be done the hard way.

Genuine question - what do you find convoluted/hard about them? To me, the apparently-industry-standard of "push a change to your App Code, which triggers a build to generate a docker image, then trigger an automatic commit containing that Docker image to a Deployment Package, which is picked up by your CD system and creates a deployment" is way more convoluted. Having a conceptual "pipeline" built out of lots of little disconnected GitHub Actions (or whatever) is also way harder for me to wrap my head around than the CDK definition of a linear pipeline.

The problem was caused by a forced deprecation of NodeJSFunction in cdk. Which basically made it impossible at the time to add any dependencies… there was nothing me or the more senior engineers could do to solve it.

I figured a work around that involved having a separate manifest for the deps and packing them manually. It worked… I also tried the lambda without any dependencies and they the lambda’s dependencies were available in the instance even though they were not listed anywhere.

The needlessly convoluted part is getting a NodeJS function into prod, the forced change caused something to break even though I was already on 18_x

But I will not lie, bats was super useful when debugging another engineer’s build, and avoiding the whole push to debug is so so so so helpful.

For the pipeline thing, idk why it was setup as such; but it certainly broke everything once a push to mainline was done, kinda like a runtime error when it should have been a compiler error. Though it was certainly on us.

CDK is fine too. Tbh I kinda love CDK and don’t want to go to anything else when it comes to cloud deployment.

In retrospect, perhaps that one sour experience had just too much of an impact.

I want a better way to manage external dependencies mostly. Afaict for scala I will need to pull the package into Brazil to get it to work. An analogue to NPMPM would be great but I can see why there isn’t one yet. A colleague had issues getting python dependencies to work so yeah.

Here’s to hoping Peru would fix those.

> I want a better way to manage external dependencies mostly

On that point, I'm totally with you. The excessive caution about the software supply chain is probably justified given the impact of a potential incident, but certainly frustrating for the >99% of times that dependencies are safe.

Microsoft has not implemented a monorepo.

However, it's not surprising when you consider the massive breadth of software that Microsoft builds, as one of the oldest and largest software development orgs.

Monorepos solve a very specific set of problems that ~ 0% of companies have.

I doubt single digit percentages of, say, the fortune 500 use them, and I'd be surprised if double digit percentages of companies use them at IPO.

Well, they do solve a bunch of problems every company has, but introduce another set of problems those companies didn't have until that point.

As with anything, striking the balance and finding when to use a tool is the hardest question.

Amazon (or, at least, my corner of it) still hadn't when I left ~9 months ago - and I'm glad of it. I've moved to a company where one of the core products (though, thankfully, not my team's) is in a monorepo, and from everything I've seen it looks like a horribly inefficient way to develop.

Why should companies jump on the latest fad and join things that don't belong in the same repo?

Isn’t this how multi-armed bandits work?

algo 101, but I can see how it can be nifty for $internalapp.

Yikes!

FWIW, I think best practice here is to hardcode all feature flags to off in the integration test suite, unless explicitly overwritten in a test. Otherwise you risk exactly these sorts of heisenbugs.

At a BigCo that’s probably going to require coordinating with an internal tools team, but worth getting it on their backlog. All tests should be as deterministic as possible, and this goes double for integration tests that can flake for reasons outside of the code.

But then you won't catch the bug before it hits production :)

Also you end up with some strange long term test behavior. Because people will often leave feature flags in place long after full release (years sometimes), you end up with a default-off-in-tests only testing behavior with everything newer than N years since the last feature flag cleanup disabled.

Yes it's kinda fractal of bad practices that have to align for this problem to occur, but that's the nature of tech debt.

You are both misunderstanding the post.

He’s not saying to alter any of the feature flags used for the test, but simply to record which were used during the test.

Simply logging doesn’t introduce any of the issues you are describing.

Huh? This is what yojo@ wrote:

> I think best practice here is to hardcode all feature flags to off in the integration test suite

That's pretty clearly about forcing the flags to be off, i.e. altering them, and not about logging their values.

Agreed, I am advocating for deterministic behavior for all feature flags in the test suite.

If you’re testing a new feature, you should have explicit tests for the enabled state (along with existing tests for the disabled state).

If you have bugs propagating up the stack from flags changing in low-level dependencies, the change to the dependency is probably not properly tested.

Alternatively, if the feature flag gates a change to the interface of the dependency, you should have explicit integration tests covering the systems on both sides of the change.

I agree that this is a real and separate problem, but I believe the solution lies outside of the test suite.

One way I have seen this handled is to enforce restricting rollouts of a feature flag to 95% at most. That way turning a feature all the way on requires removing the flag from your codebase. It’s draconian, but honestly anything less than that leads to the situation you describe.

I like that idea a lot. We've been informally doing it on my current team, made easier since we can sort of cleanly do atomic code+flag updates in a single commit

No, the best practice is that on each test run, every feature flag used implicitly or explicitly needs to be captured AND it must be possible to re-run the test with the same set of feature flags.

That way when you get a failure, you can reproduce it. And then one of the easy things to do is test which features may have contributed to it.

I strongly disagree. If you have non-deterministic tests, you are going to have builds breaking for unrelated changes, seriously hampering developer productivity as teams chase down failures unrelated to their change.

Nothing kills confidence in testing more than test flakes. It’s a huge drain on velocity and morale, and encourages devs not to trust test output.

If you want to have some sort of chaos monkey process that runs your test suite flipping feature flags at random and notifying teams of failures (along with some sort of resourcing to investigate) I could get behind that. But that should be something outside of the main suite that gates code deployment.

If a test passes when run by a dev pre-commit, it should pass in CI.

Man, this story sounds like you could be on my team :-) Pretty much experienced the same stuff working at BigCo!

In the end, I think the real problem is that you can't test all combinations of experiments. I don't trust "all off" or "all on" testing. In my book, you should indeed sample from the true distribution of experiments that real users see. Yes, you get flaky tests, but you also actually test what matters most, i.e. what users will - statistically - see.

This sounds like a situation that would benefit from using an approach like all-pairs testing - https://en.wikipedia.org/wiki/All-pairs_testing

Basically, if you have N different features (let's assume they are all on/off switches, but it works for multi-values too), in theory you'd need to run 2^N tests to cover them all, which would become completely impractical. But, you can generate a far, far smaller set of test setups that guarantee that every pair of features gets tested together. Run those tests and you'll probably encounter most feature-interaction bugs in a much quicker time.

All-pairs is for _pairs of features_. For subsets you're in much deeper trouble because of the exponential dependence on N. For a fixed polynomial dependence, you can get clever and let tail bounds eventually work for you, but for exponentially growing hypothesis sets, that won't work.

I don't understand what you're saying. That it doesn't work if you need combinations of 3 or more features (a subset?)

This is relatively subtle stuff, but here's an attempt at describing the general problem. I'm going to describe the deterministic case, but the probabilistic case is effectively the same.

Let's say you have a bug you suspect is from an interaction of any one pair of 10 features being "on" or "off", but you don't know which specific pair causes the problem. Encode each of the states you could set up your code by a 10-digit binary string: 0000000000, 0000000001, 0000000010, 0000000011, etc.

We could try the 45 possibilities in some order, and we would expect that on average it'd take us 22.5 tries to find the bug. But notice how your "target set" is smaller than the universe of strings: there's only 45 pairs of features, but 1024 strings.

What happens if we try a random string of ones and zeros? Now, instead of catching just one possible pair, we are covering many pairs. The only problem is that we now won't be able to know exactly which pair caused the problem when it does. But we can build a corpus of strings that don't trigger the error vs. strings that trigger the error, and a random sampling soon converges on the correct pair.

If you think about why this works, it's because any of these random strings has about a 1/4 chance to trigger the bug: wlog we can reorder the bits so that the buggy feature are the first two digits, and then we see that we have a 1/4 chance of hitting "11" on those two digits.

The problem is that as you increase the size of the subset that needs to be active, the probability that your random strings will actually catch the bug decreases exponentially. For any _fixed_ target size k (the number of features that need to be active), the overall complexity is still polynomial in n (the number of existing features). But if k is a constant fraction of n, then this technique takes exponential time in n.

Fun, thanks.

I wonder how well letting a fuzz tester generate the test configurations would work.

Probabilistic feature flags! Love it.

Always base the probability on something stable, such as hash of the username.

Bug report: changing my username breaks $product.

Yeah no thanks. It's probably better than completely random but software should be predictable and unsurprising.

The important part is the stability - if your usernames can change then they aren't stable so you don't select it.

I think it is a good reminder that most things you think of as being unchanging that are also directly related to a person.. aren't unchanging. Or at least any conceivable attribute probably has some compelling reason why some one will need to change it.

> changing my username breaks $product.

https://m.youtube.com/watch?v=r-TLSBdHe1A&t=14m10s

Discussing a performance regression due to longer username due to username being in ENVIRONMENT variable which changes memory layout of process.

I've used the hash of username+string trick before for a flag. I used it to replace a home-grown heavyweight A/B testing framework which had turned into a performance bottleneck.

It worked quite well.

That's why you have internal user ids instead of using data directly provided by users.

Will it cost an extra lookup? It's cheap, and if you really need to, you could embed the lookup in some encrypted cookie so you can verify you approved some name->id mapping recently without doing a lookup.

Wait, we're talking about maliciously injecting bugs into your employer's software so they have the maximum impact, right?

Clearly, making sure that 1% of all teams gets fired for being unable to run unit tests, then slowly ramping that by a few percent each review cycle is a good strategy.

Ideally, the probability of breaking would drop off exponentially as you moved up the org chart. Something like "p ^ 1/hops_to_director_of_engineering" would work well. The trick would be getting the dependency to query ldap without being detected...

This is about a feature flag. It should be safe to flip it on and off, otherwise rollouts and rollbacks won't work.

You just need to be able to look up what feature flags where enabled on a given request (maybe by correlation I'd)

Or have the username be a number that is all the feature flags when converted to a binary representation. Then you can just have one username for each combination you want to test.

You just need to make sure that this doesn't mean people are consistently "lucky" or "unlucky."

I was on a team where app updates were deployed using a canary system. A small percentage of users (say, 1%) received the update first, then the team watched for incoming crash reports from that cohort. If it looked good, the feature was rolled out to a few more people, and this was repeated. This allows you to identify a problem by only negatively impacting a relatively small percentage of customers.

The problem occurs when the calculation to determine which cohort the user belongs to is deterministic. In this case, the calculation was based on the internal ID of the user. This means some users always get the updates first, and deal with bugs more frequently than other users. Conversely, some users are so high in the list that they virtually never get an update until it's been tested by a wide user base, so their experience is consistently stable.

Or you might have a problem where some players in a video game consistently take more damage than their friends: https://news.ycombinator.com/item?id=34742505

Multi-armed bandits utilize this

This is my daily life at BigCo. These bugs are the worst.

Eh this is what version pinning is for. Using edge can always lead to random breakagr, feature flags or not.

That wouldn't work here though, the dependency started by breaking an almost-undetectable fraction of the time.

Imagine a scenario where your upstream dependency started out with one failure per 1,000,000 machine hours, then removed a zero once every 12 months. If you had 100 machines running tests at 100% efficiency, the bug would hit about once a year for the first year, then 10x the next year, and so on.

Put another way, if upstream is malicious, and you're not auditing every line of their source code, you're screwed.

That depends whether the development model is branch-stable or trunk-stable.