You've heard of “yes men”. Now learn about “no men” (2011)
rachelbythebay.com
rachelbythebay.com
It's 2019 now though. only 8 years later, but a lot of things changed.
The industry is growing fast, faster than knowledge can be shared even with all the power of the internet. It's very bottom up, with newer engineers severely outnumbering those that have been around. People are making fat salaries and companies are catering to their whims. They/we're more spoiled than they've ever been. People go to work hoping they'll be find a reason today to try the latest pattern they saw on Twitter or try the trendy new tech or library their friend just talked about.
If you don't have people saying no, it quickly (very quickly) get out of control. As organizations grow, you need to start saying no. A lot. Else within 0.2 seconds of the new intern joining or the new grad coming in, or the lateral hire who was the king of their 5 people startup will be pushing some stuff that isn't mature enough or doesn't follow established patterns. More often, they'll be reintroducing something problematic that you just finished getting rid of 6 months ago.
It sucks. Sure, there are some power hungry people who get off on power trip by being gate keepers. But overall, it's just people desperately trying to keep things in one piece and the company going, and that means making hard decisions and enforcing them. It just doesn't feel good to say no, arguing all day with people who have way more energy than you do, but it has to be done.
There's a big gap between that and fighting cargo culting.
Where it is necessary to change (or to make changes) without full understanding, at the very least, instrument and observe for some time.
Aping someone else's fancy, new solution is not the status quo. You have no way of knowing how badly they have steamrolled Chesterton's Fence.
Same thing with auto formatters and linters sidestepping a lot of pointless bickering and bike shedding.
As I mentioned in another post, as your org grows, there's going to be an infinite amount of things people push, and pushing back is hard. If you have to stop and write a consensus document for every single one, you'll be doing nothing else.
spoiler alert: it isn't hard and this is what we do at my startup that is currently experiencing hyper growth and is on multiple "unicorn" lists
If you have the authority to tell people no in the first place to make it stick, you have the ability to write that in a wiki page. You can go full FAQ mode for the miscellaneous stuff as it comes up.
Documenting the why is really important, so you don't get zombie decisions self perpetuating in your org and allow for change.
And they may never get hired and you still have to fight with them.
I'm, honestly, TERRIFIED of what "thought leaders" may post on hacker news/twitter/reddit tonight, tomorrow, next week. The moment I see one of them suggest something that just wouldn't work for us, I -know- I'm going to have to fight 50 people at work soon who drank the koolaid.
When something isn't personal, it diffuses a lot of emotional tension. Our monkey brains jump to taking things personally way too easily probably as some sort of early tribal instinct for survival.
I don't really understand what you mean by personal otherwise? I was implying 'personal' was specific person personal.
javascript
curried functions
We started to wonder if his blocking of everything was just pathological because the tech stack wasn't all that great.
Otherwise you're kind of a dick, and nobody wants to work with that. Unless you're in charge and they can't get away, I guess.
And it really was an explanation, not a condescending or dismissive refusal. That part is key.
I was involved in the design of an ASIC. This was going to be used by people from all different areas of the company, and they all had various demands. Most had never worked on a chip before, and didn't realize the cost of what they were asking for. The easy thing to do would have been to ensure that every requested feature was added But that would have burned area, power, and huge amounts of design and verification time.
Luckily we had a senior guy who had the clout to say "no". Over and over again, he would tell people how much things would cost in terms of money and time, and ask if the features they wanted were worth that much. Or he'd point out that what they wanted could be done with firmware plus existing features. Overall, he probably saved millions of dollars and caused the project to be delivered on time.
Do we always listen to him? No. Does he always win? No. Is he always right? Absolutely not. But I keep him around because he forces everyone in the company to defend every idea. If you still feel good about the idea after defending it from him, then go ahead. If you've lost faith, then I'm glad we didn't waste any time on it.
Imagine how much better the Star Wars prequels could have been if only Lucasfilm had a strong No Person to stop them from doing stupid shit.
Arguments put forth should be criticized and challenged. However this becomes difficult when egos come into the picture. There are stubborn engineers that will never accept that the possibility that they might be wrong. With this in mind, the "no man" cannot be the decision-maker, the "no man" should be playing the role of a lawyer.
I've learned some people can take personal offense to having their ideas challenged, particularly when...ahem... There are some glaring holes in them they frankly should have caught. So over time I've found ways to try and take the people out of it, and also to be enthusiastic about the act of having ideas, because I'd rather have people have a bunch of ideas that mostly don't work, with occasional nuggets of gold than people who aren't capable of or engaged enough to think of new things.
Personally, I start considering most ideas, particularly my own, from the standpoint of "why will this NOT work?" It helps me sift through things quicker by identifying potential deal breakers earlier in the process, and helps me make sure I've done my due diligence on important decisions.
The net result is that I kind of enjoy poking holes in things all day, even if others see it as "not optimistic enough."
There's also a time and place for experimentation, and a time when it's got to be made clear that there's only so much time (or other budget) for research, and that that's going to be tracked and burned down (though the discretion may be put on the R&D team itself to decide how to allocate that).
Much of Rachel's essay is more about management-by-mythology than about the saying-of-nos, specifically.
1 - there is a Mr.No guy but, the author only knows him by reputation
2 - there is a technical problem that seems to revolve around memory
3 - the author has an idea of a solution, she actually tries as people don’t believe her
4 - her solution works, but she is told the “No” guy said “no swap” once a long time ago
Her reaction: “Did I care what he supposedly said? No.”
Why doesn’t she go talk to him ? The “No” guy could have had another solution than raising swap. He could have recognized being wrong, allow swap and learn from it. He could continue say “No” but now the author has an actual experience of it, and not just hearsay and reputation.
It seems like nobody communicates correctly in that company, the author included, and it must hurt in so many places.
Edit: fixed author’s gender, thanks
The chances of that are astronomically low. From the article:
>I later found out that all of our older machines had swap enabled. It was just the ones which had been replaced or reinstalled which had been popping up without it, perhaps as part of a policy change. That means it was perfectly valid for us to run with swap on, since most of the machines we owned were using it.
(I fixed the remaining bits)
I mean... it was an interesting idea, writing a book that way because the characters sexes shouldn’t really matter. But in practice, it was a mess.
Depending on what you think of Google (or maybe Rackspace, but I think Google's more likely the source of that post), you might be surprised at what company you're talking about.
I think you're spot on but am curious if you'd write that if you knew where they worked.
It would be less surprising to have these kind of disfunctionments within giant corps I think. As a consumer, Google's product strategy also mirrors that image (the number of messagings platform they released that ironically don't talk to each other is legendary by now)
In particular, a very small company with a lack of internal communication would fail very fast, so it's less likely IMO.
Put the responsibility on yourself to make the decision instead of passing it off to others.
2. Said problem was OOMs on machines with swapoff
3. Yup
4. It's pretty easy to get someone's measure from reputation if you have multiple independent sources - enough to fully distrust Mr. No's judgment. If this person is high enough in the chain that she's getting his orders second-hand, it also may not be very easy or fast to get his attention or a face-to-face meeting. So she made her change, publicly stated she was making it, and allowed him to countermand her (with a snarky "he can have my pager" to force him to justify his decision if he wanted to reverse the decision).
At least with the swap example, there was someone you could talk to, or ask your manager to talk to. But what if it was just an Onion in the Varnish and nobody remembered who said no swap in prod, it was just "That's the [company name] Way"?
After a while, those two leaders left, and we started seeing issues that the policy was causing. However, people still stuck to the policy. When I would argue that the policy wasn't really accomplishing what it was originally intended to do anymore, I was repeatedly told it didn't matter, it was the policy. There was no one to even argue against anymore, because they were gone, but the policy remained.
I was there when the no swap policy was under lots of active discussion. It was there to ensure that application authors could have strong expectations on how long a memory access would take. Imagine trying to provide evidence that removing the "no swap" policy wouldn't affect any of the applications that were running across a very large infrastructure.
Her point really resounds though, this person was referred to all the time in conversation, "well, we probably couldn't get this past X". He was mythical. In general that was often a good thing. Making changes that could have such a broad impact should be well reasoned. But it's also quite annoying, having a nagging voice inside your head that just says "no", seemingly disconnected from the actual source.
"No swap in production" probably stemmed from someone who worked on the servers, and that person discovered they were swapping too much, which would eventually lead to hard drive decay. And, maybe for budget reasons or whatever, they decided to make it a policy to disable swap, rather than increase RAM or change the underlying software that caused the servers to use too much RAM.
And now, later, on different servers with different software and different specs, the policy still exists, because nobody really paid much attention to why it was created in the first place.
Like everything swap has to be monitored and there’s no need To disable it entirely
I've encountered something of a converse of this situation: a set of prod servers running a Java workload on which performance was frequently abysmal, which turned out to be due to a very poorly tuned set of JVM GC parameters. The feature(s) were poorly documented (lots of detail, little useful guidance), so it took some tuning and ongoing performance monitoring to see that 1) the problem had been solved and 2) that no new problems were introduced.
The latter being of particular concern as this was a long-lived project that had gone through generations of development and ops support (I was latter), few of whom were on current speaking terms with the shop in question.
The management feedback I got on my monitoring was (a rudimentary bit of instrumentation, and periodic checking) was that this seemed to reflect an excess of concern and/or attachment to the issue and/or solution.
Given the numerous other years-outstanding issues which I'd identified and (eventually) largely remedied, I found this response tremendously perplexing.
If enabling swap 'fixes' it, isn't the proper fix to provision more memory?
Mr. No says all servers must match one of the existing configurations, none of which have very much ram.
If you're on the border where a little bit of swap gets you zero pages, and no swap gets you stuck machines, you probably need to figure out how to scale within your constraints, but a little bit of swap should be ok, in the neighborhood of 1GB gives you a little bit of breathing room without prolonging the inevitable for very long.
2. At some point you do have to change configuration, and if you have insufficient memory on one you either have insufficient memory on all machines, or a badly distributed workload.
This program uses 1 TiB of virtual address space, and runs fine on a machine with no swap (and much less than 1 TiB of physical RAM).
#include <sys/mman.h>
#include <stdlib.h>
int main()
{
size_t const len = 1024 * 1024 * 1024; // 1 GB
for (int i = 0; i < 1024; ++i) {
char *p = (char *)mmap(0, len, PROT_READ | PROT_WRITE, MAP_PRIVATE | MAP_ANON,
-1, 0);
if (p == -1) exit(-1);
}
}
I wasn't precise enough. The case where we needed swap not to error out was in a big mapreduce process, consuming maybe 44GB out of a 48GB host. When we didn't have swap on the box, the big process would exec a small command line process and the exec would fail, there wasn't 88GB of address space available during the brief period between the call to fork and the overlay. Adding a swap device fixed our problem even though it was _hardly_ ever used (MB to single GB).
At least that was my analysis from 10 years ago. I never went full science on it, but I did read plenty of documentation on overcommit and virtual address space.
echo 1 > /proc/sys/vm/overcommit_memory
Edit: It appears from https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/lin... that things will go wrong if you try to allocate more VM space in one call than the total free RAM+swap available to the system.
When redis is persisting its state to disk, it temporarily needs 2x the memory because it forks a child that does this in the background. Since fork uses copy-on-write, only the difference between child/parent is actually consumed as extra memory. The rest is "de-duplicated" in a sense.
The choice is then to
1) enable swap, possibly killing performance 2) enable overcommit, possibly killing the redis process
It is sometimes preferable for a service to fail hard and fast than get bogged down with swap. I believe this is the idea behind the "no swap" policy, at least on Linux.
Redis is a good example here because any amount of swapping will kill performance since literally all of its allocated memory is consistently accessed for the purposes of doing a dump. And Linux will always try to swap things out, even with vm.swappiness=1.
I don't believe this "no man" is someone who is managing their attention or saying no for an actual reason. They're just contrary. And yeah, there's a lot of them around.
What I actually find even more frequently are "parrots". These are people who "learn by runbook". Instead of asking why something is done a particular way, they want to learn how to parrot some procedure. It's another variation of things: instead of saying "yes", or "no", they just parrot "process X".
I think this is really where the "no swap in production" example came from. I've seen _lots_ of stupid similar ideas in software development.
Usually, they are in the form of protectors of the "process". This wasn't vetted by so and so, or didn't go through the proper channels, or doesn't use the right technology. And 51/100 times, they may be right - they keep unnecessary scope creep, bad ideas, technical debt, overhead out of the process. Literally the only reason people bother following a process is because of them.
But at the same time they are a huge drain on time and resources, and can enable bad or inefficient processes by manually keeping things in line. They add nothing to innovation - and it's a huge bummer to work with/around them.
Most process is scar tissue from something that went wrong in the past. We have mandatory code reviews because when we didn't we let a severe security bug get deployed to production. We have legal review all changes to the terms of service so we don't get successfully sued over it again. We have a schedule with code freeze N weeks before release because when we didn't we could never release anything because...just...one...more...change.
I was hired at a small company once without any process for anything. No release process or schedule. No bug tracker. No testing plan. No code reviews. Nobody used source control. Builds from different workstations produced wildly different binaries. It was a total madhouse and it was amazing that anything got done at all. The CEO simply thought the engineers just weren't competent enough, but they were--it was just a circus because nobody stepped up to formalize anything. Once we got a bug tracker and SVN set up, things turned around pretty quickly.
Eventually I was dismissed from this company a couple of days later and this situation was raised as one of the points. I was told "you can't refuse a mission given to you by your manager, regardless of the circumstances, that's ridiculous" and when I replied "I have different views to you on this" it was considered to be rude. The person who dismissed me is more or less the management boxing bag - a certified "yes man" who nobody respected.
Of course this hasn't changed my attitude, though. Still a proud "no man" and wouldn't go with the flow unless I'm sure everyone completely understands the consequences. At some point people have to realise that their loyalties don't go towards their manager and instead should go towards the shareholders and clients. There is a bit of give on this, of course, but eventually you have to do the right thing by the people that actually stand to gain and lose - not just being another cog in the machine that exists only to support other cogs in that machine.
Why your manager is making technical decisions is beyond me as well. Managers are supposed to be managing people, not doing engineering work.
Besides, come on, who even has dogmatic rules like "No Swap In Production". That's just short-hand for a concept. When it's in dispute you have to trace the reasoning again. There's no alternative. I cannot imagine doing something only because someone said so. I certainly recall a time when someone said "Person X isn't going to like this" and saying "Well, then, person X is going to have to find another way to solve your problem". I certainly hope that if I'm person X in someone else's story, they take the same approach.
And if you have to trace the reasoning more than once, it's probably worth writing down your argument. Not as a big RFC-style thing necessarily, but even just a Slack thread you can point to.
Think of them as real-time meeting minutes and it may be easier to swallow.
This is reinforced by the additional fact that the people with the most I have to learn from are the people with the least time to teach me. So I minimize the effort I require of them to give me their best.
For instance, if I had a chance to ask Feynman stuff, I would happily accept answers in txtspeak with emojis and broken grammar. Sort of a personal Postel principle.
You’re also on the hook for impeding the organisation.
I’m a “no” guy. My boss is a pure yes man.
I don’t say no to impede progress. I say no to avoid overworking my team which would reduce the quality of our work.
I say no, because it changes the scope of work in progress, and that requires sufficient justification because it means either throwing work away (which is both demoralising and wasteful), potentially increasing time to delivery or creates technical debt in droves.
I say “no” so that people don’t make their own standards (like not following logging patterns, or metric naming conventions) which would cause all kinds of gunk in the infrastructure.
It’s not about following a strict process. Although that can be important for streamlining workflow through an understaffed team like mine, it’s about ensuring that the quality of what we do is not at risk.
Quality is our only currency, without it everything suffers.
If one needs a mental picture, think of the quintessential hipster:
https://sabotagetimes.com/life/hipster-irony-is-destroying-o...
But there are times, such over ADA/accessibility and data security issues where I am somewhat insistent, but people ignore me. They seem to value a "cute" UI over these. I'm doing the right thing but get rebuffed. We are not in the cute-UI business. Our domain is not a toy or fashion store. We've been sued for ADA issues in the past, and our competitor has had data breaches, so they are relatively likely risks. It's frustrating to be dismissed the way I am.
Most organizations fail horribly at saying no. Instead they try to silo things to cope. This works to some extent, and you will always have silos as a company grows up, but you really should never end up with 12 programming languages, 5 RDBMS, 4 NoSQL DBs, 3 Javascript frameworks, 3 CIs, 3 cluster schedulers and then all the aforementioned also duplexed across organization units. That level of sprawl reduces margins, increases onboarding time, stresses even the most experienced staff, and makes adaption and business decision making murky at best.
> "so-and-so says no swap in production".
That said, while I understand the point of the article is something different, I broadly agree with this advice: generally no swap in production.
If you need swap, then that's probably a sign that the workload isn't matched well to the resources of the machine (unless the workload was specifically designed to employ swap, which most are not).
The problem is that: maybe most of the time the application's working set will fit within memory, and so it will perform well, but some of the time it will begin to swap and performance will degrade. The result of adding swap to a system is non-deterministic performance.
If you're building high-availability, high-performance production systems, then you don't want the potentiality of arbitrary performance degradation or non-deterministic behavior. Instead, you would prefer to solve the root cause -- such as by increasing the amount of memory available to the process or system, so that swap is no longer required.
An engineer adding swap to a system is like a doctor treating a patient's symptoms instead of curing their disease. Both are useful but one is preferable. If you encounter a situation where you could solve an application problem by adding swap, then an even better solution is to address the root cause -- ensure the application doesn't run out of physical memory. Swap is a band-aid that masks a risk better addressed directly. Hence: no swap in production.
(On the other hand, it might be fine to employ swap in production for services that aren't performance sensitive, and where it's preferable to accept the performance hit from using swap rather than pay more for better hardware. That's a fine trade-off if consciously made. In my experience, variations in performance are usually much more problematic to a business than changes to cost.)
Sometimes, but sometimes you want the opposite.
A slow service (e.g. a db server with a failing disk) might still pass health checks but degrade overall system performance.
It is easier/simpler to detect a total outage than a slow/flapping service.
This is a bad policy. The problem is that it breaks Linux's overcommit logic. Many applications allocate huge chunks of virtual memory far in excess of what they actually use. Due to the magic of virtual memory, this doesn't get mapped to physical memory until it's actually used.
The trouble is that there's no mechanism in any programming language I'm aware of that allows the OS to retcon an allocation. The only tool the kernel has at its disposal is the oom killer. So if you allocate too much, and you have swap disabled, the oom killer has to step in and kill something even if you're only using half your ram.
This is bad enough, but in addition to this, you have vfs cache to worry about. If you have 64GB ram, your applications are using 40GB, and you have 22GB cache, you and I might assume you have 24GB free, but the OS only sees 2GB free, and a few allocations might trigger the oom killer if they happen faster than the vfs can dump cache. (which is surprisingly slow) With swap, this isn't a big deal, because the pages don't actually get assigned until they are used, which happens later, so the vfs has plenty of time to evict.
The point of swap isn't to give you more memory. It's so the OS can keep its promises about memory allocation and overcommitting.
You can allocate all the memory you want, the OOM killer only steps in when you actually use all the the memory.
If you don't care, it's much easier to default to someone else's directive, or complain about a toolset you haven't bothered to learn.
btw, it was /proc/<idiot's username>, not /dev :)
I've met "Mr No" at Google, and he is in fact quite a reasonable and funny guy.
Can you recall what this code you wrote previously do? “I can’t remember.”
Do you know where the files are placed? “I don’t know.”
Where are the documentations for xxx. “I don’t know.”
Do you know the difference between xxx in testing in production server? “I can’t remember.”
It was frustrating.
9 out of 10 questions were “I don’t knows” and “I can’t remembers”.
I’d much rather he said yes or no.
> Can you recall what this code you wrote previously do?
What it actually does might be different from what I remember. It might have been changed since then -- maybe by me, maybe by someone else. I might have made a mistake in the first place, and it doesn't do what I think. "I'm not sure, let's look at it."
The benefit of domain aware lawyers is they can explain the specific risks and help the decision makers apply those risks to the business.
> Reagan used the phrase to emphasize "the extensive verification procedures that would enable both sides to monitor compliance with the treaty"
It’s obvious when applied to trivial situations:
“I trust my wife, but I take the time to verify her phone every night and check the private eye report of her day”
We wouldn’t call that “trust”, pretty much the opposite
Gotta love simple fixes.
The thing that the maybe guy differs from the Yes or the No guy is that he's always 100% correct. He's never wrong ever.
Is this, then, Chesterton's Berlin Wall? That is, a completely idiotic waste based on idiot politics?