Hacker News Reader

Comment by iamdave | Hacker News Reader

Slightly off topic but articles like this make me weep as I look at my infrastructure and think about all the best practices that we by all means should be following, but aren't because of a senior engineer stuck in the 1990's and is seemingly not only paralyzed by analysis (and bikeshedding), but seems to subscribe to the notion as a way of living, and management who defers to him at every turn despite all evidence of how it holds us back (not to mention directly contributing to outages). I'd give my first born child for immutable servers.

Consistency in database systems refers to the requirement that any given database transaction must change affected data only in allowed ways. Any data written to the database must be valid according to all defined rules

I would LOVE for this to be a thing. Just as soon as I can convince people that maybe storing my.cnf in VCS is a better idea than updating it in prod. Live.

Sighs.

View on HN

>I would LOVE for this to be a thing.

It is a thing, just not at that company.

Seriously, get a year or two, then look for escape options.

There's a Mt. Everest climb's worth of distance and effort between "it works" and "best practices", and like successfully climbing Everest, very few companies are doing all of the best practices, and I suspect your dude going the opposite direction by digging with a spoon.

Seriously, get a year or two, then look for escape options.

Been a year and a half. Have a 2nd round interview lined up for Thursday to get off this crazy train.

I also happen to be reading The Phoenix Project. Which is...strangely therapeutic ;)

There was a job when I was still green where they had fired an architect and they were bringing in more hands on people with a hope someone would step up. Someone opined that architects have a half-life of about 12 months. I ended up splitting lead duties with another person and she and I split the architectural oversight.

Since then the same patterns (alas, minus the diversity) have worked out a number of times and I’m adamant that adding another lead is far better than consolidating that under one person, at least until you have half a dozen leads.

In fact my most useful mentor and boss said to me look, I trust you guys to sort this out. I want to hear the plans and ask questions but I only need to be involved when you can’t come to an agreement. And then I can be your tie breaker.

I think a lot of things would go better if folks in these positions would audit and break ties and worry about bigger picture stuff most of the time.

I had a boss like this and very much believed in that sort of autonomy, only getting involved-as you said to be the 'tie breaker', but still lead a great engineering team and was a great source of knowledge and mentorship.

I say 'had' for a reason. That reason is mentioned in the post that started this comment chain. Now what's that tell you? Heh...heh :(

I had a different boss who trusted us implicitly but essentially we were managing ourselves. Unlike Mr Mentor we didn’t talk shop and he didn’t challenge us very much or at least concretely to do better. In fact I was only ever aware of him doing about 20 hours of work a week and half of that was meetings (the other half administravia).

Given the politics there I suspect the rest of his time was kissing butts or covering them. Since the end of the dotcom era I’ve never seen a more high minded bunch of Emperors New Clothes types all in one spot. Woof.

> I also happen to be reading The Phoenix Project. Which is...strangely therapeutic ;)

Huh, I started reading that earlier today. Between the book and replaying levels in Infinifactory, the words "bottleneck" and "backpressure" loom large in my mind.

Some people get ahold of tunnel boring equipment and go as fast as possible in the wrong direction. With this mountain in the way we can’t tel if there’s anything cool on the other side, but damn are we hopeful.

Not really related to your main point, but speaking as a greying "old guy", one of the things that's really hard is to gracefully allow the next generation to contribute. I still vividly remember starting out and being incredulous about some of the stupid ideas the senior guys on the team had. I was proven right quite a lot of the time too. It's easy to build that inner narrative of the hero righting wrongs.

But then fast forward to the back end of your career and you've got a kind of weird dilemma. The vast majority of ridiculous things that the young guys try to push is absolute crap. They read about it in a blog post somewhere and don't have nearly enough experience to understand the nuances. Or they have yet another stupid f-ing framework that they just know is going to revolutionise the industry. Yeah... been there, done that, my closet is overflowing with T-shirts, thank you very much.

You sit there thinking, "I spent 20 or 30 years fighting my way through this crap and when I finally have a good handle on it, these new guys are just going to throw it all away". But... Dammit. You absolutely know that one of them has a great idea and you're not quite sure which one of them has it. Because they all think they have it sorted (even when none of them do). They feel they are completely sure that "this is the best way and only a moron couldn't see it". But you know that it's always more subtle than it appears. If you let them all run with their ideas, then 90% (99%???) will crash hard... Let's face it -- just like you did when you had that stupid idea that you were absolutely certain was "the way forward".

Anyway, I belabour the point. From my vantage point, it's pretty difficult to let go. Some people just can't do it. In my position now, I have a lot more sympathy for the "old guard" than I did when I was younger. To be honest, I think it is technically impossible for me to have less sympathy than I did -- and I think this is pretty common in the industry. Probably both sides of the equation are in a similar situation -- neither side feels comfortable letting the other side make decisions.

Just talking out of the hole in my head right now, but I think the main thing is that programmers generally like to program. They like to have freedom to try approaches that they think will be successful. As much as possible, it's probably best to allow your teammates, whoever they are, to have the freedom that they need. At the same time you should expect to have your own freedom. Getting the balance is not easy and sometimes you have to change jobs in order to get into a team where you are allowed the freedom to do your best. In my experience, teams that take this seriously are the best teams to work on (even if they sometimes do stupid things).

You know, I really appreciate this response and perspective; because it makes total sense. It's human to want to stick to what you know and only natural to be hesitant of the unknown, especially when you have the tenure to have seen it all over the course of a career. That experience is valuable and it's why in many cases I do lean upon this senior guy for input on specific topics I know he is the best resource for.

That said, in this specific context of tech (or at least in my wheelhouse of Ops), this is surmounted by creating and fostering a culture where it's okay to experiment and fail.

Where we've been handicapped in this regard is specifically not doing that, and the particular individual who makes this difficult having the mentality that any new approach spells doom and actively working against initiatives to put us in a better posture of experimentation in environments that don't sacrifice stability and reliability of our stack (hence the sarcastic comment about giving up my first born child for server immutability--I don't even have kids, haha).

But that said, I think you're absolutely right. I'm not quite greybeard yet-mostly because I yank out every grey hair I see trying to establish residency on my chin, but nonetheless your point is accurate, well taken and definitely that has immense merit.

And if you allow experiment and fail, you end up with 20 different frameworks in your app.

And that means a huge cost of inconsistency. Every time you need to make a little tweak you have to learn how that part of the app works.

When you just want to get shit done, being able to go into any part of the app and knowing it's built exactly the same way as every other part means you don't have to spend any time learning insert X framework, you spend time doing, implementing business rules and fixing bugs.

A lot of programmers want to spend time playing with tech, not doing because playing with tech is more fun then doing the job.

Worst still, juniors (including me back then) are woefully unaware of just how clueless about programming they actually are, as alluded to by the parent, and that means they have massively over-inflated opinion of their views. They also tend to be wildly optimistic about how long a change will take and totally disregard the human cost (the cost of re-training and loss of productivity) of changing a piece of tech.

This is something I think about a lot nowadays. You have to try and fail a lot to develop an intuition for good design. How do you give people the room to experiment without paying for all the mistakes? It is tremendously expensive to produce a senior developer yet vital to do so because junior developers mostly produce crap. We could only assign them projects off the critical path, but we can't really afford to have anyone working on anything but the critical path. Plus it's a lot to ask of someone's ego to say, here work on things that don't matter for 5 or 10 years and we'll talk.

I was randomly watching Youtube the other day and there was an ad for Herbie Hancock's Masterwork class (ha ha -- not a referral, no idea if it's any good). Anyway, in the ad he mentioned that he was playing with Miles Davis one time and that he made what was technically a mistake. Miles listened to it, picked it up and made it right. Suddenly I realised: "That's it! That's what I want to do." But, of course, it's much easier said than done. I won't pretend that I'm able to do it yet :-)

You do it by modularising code into smaller parts that communicate in well defined ways and enforce that. Then you can take whoever and have him make one module with a lot of autonomy. If he fails and make mess of it, someone else (or even more experienced himself) will rewrite it relatively easy.

Meanwhile, you let him make decisions in his module/plugin/part/web service/whatever while you enforce rules between them.

Also, the work off critical path matters too, it would not be done at all otherwise, so treat it like it matters. It won't take juniors ego if older collegues take that job seriously.

That's what "side projects" are for. Go and experiment as much as you want. Then after you've succeeded, make a presentation to technical management showing what you've learned.

Usually, if you define requirements you can make an objective choice. The old guard is too attached to stuff they are fluent and young people follow hype too much.

As the technical field, we do not follow the scientific method.

> As the technical field, we do not follow the scientific method.

Surely we should be looking to engineering here, not to science.

I'm the oldest guy on my team and have an architect title, yet I'm having a very hard time getting my younger colleagues to adopt CI/CD, microservices and agile. Some of them know what they are and that they sound cool, but don't know how to do it. Some of them actively oppose it.

"Sound cool", do you have a better reason to change the way your colleagues make a living other than it sounds cool?

I'm saying the team knows they "sound cool". I have lived them all and know exactly what they're good for and why.

This is ageism plain and simple.

You clearly have no clue what was the predominant view in 1990s. Consistency wasn't just _a_ thing in the 1990s. It was a core dogma even beyond database systems. It was top of the agenda for a while until it sort of fizzled out in the early 2000s when distributed transactions hit a complexity wall.

How do you reduce transactional complexity by throwing out consistency? Genuine question. At the very least, it seems like you need to mark potentially inconsistent data as dirty.

I didn't mean to imply that you could. But the simplicity that consistent distributed transactions can deliver in principle can easily get drowned out by accidential complexity created by a specific implementation.

For instance, you have to get all participants in a distributed transaction to support the same standard protocol without too much bridging, wrapping, special interface description languages, tons of configuration metadata, etc. You have to agree on a small stable set of features that everyone implements in a robust way.

But the major corporate sponsors of the competing approaches originating in the 1990s and early 2000s got disrupted before any one of them could gain enough traction to bludgeon everyone else into compliance.

My point was merely that almost no one in the 1990s doubted the necessity of consistent transactions. Eventual consistency and all the rest only became popular with NoSQL and the general desire to imitate internet giants that didn't even exist in the 1990s.

Sounds like said "senior engineer" is an example of what I discussed in response to https://news.ycombinator.com/item?id=15975130.

Don't let it happen to you.

> Slightly off topic but articles like this make me weep as I look at my infrastructure and think about all the best practices that we by all means should be following, but aren't because of a senior engineer stuck in the 1990's and is seemingly not only paralyzed by analysis (and bikeshedding), but seems to subscribe to the notion as a way of living, and management who defers to him at every turn despite all evidence of how it holds us back (not to mention directly contributing to outages). I'd give my first born child for immutable servers.

I know that feel although in my case its non-technical management who just want things to be frozen in time.

Cest la vie at some companies unfortunately. Good luck with your escape plan. :)

What does the 1990s have to do with it? The 90s were all about data consistency (probably too much so).

> all the best practices that we by all means should be following, but aren't because of a senior engineer stuck in the 1990's and is seemingly not only paralyzed by analysis (and bikeshedding), but seems to subscribe to the notion as a way of living, and management who defers to him at every turn despite all evidence of how it holds us back (not to mention directly contributing to outages)

Huh, that's exactly how I feel about the "must use a RDBMS for all data" senior architects I've worked with.

>> storing my.cnf in VCS is a better idea than updating it in prod.

Eh, 'but we can't store things/API keys/passwords in VCS! It's bad!' is a pet peeve of mine. No, let's not 'just update it manually when we need it'. No, if outsider getting your DB password (somehow) poses existential threat... Chances are that you have way more concerning problems.

Storing passwords and other secrets in VCS is a phenomenally bad idea, particularly when your team size grows past two or three people.

All it takes is one fuckup of accidentally committing to a public repo. Or firing someone on the team but not thinking to rotate secrets. Or hiring an outside contractor that now you implicitly give access to your production AWS credentials, etc.

Secrets do not belong in source control, hard stop.

It is wholly possible to store configuration data in vcs in a way that doesn't sacrifice depth of security when it comes to secrets and passwords.

But otherwise yes, storing passwords in vcs is bad. That wasn't necessarily what I meant to imply by suggesting storing configuration files in vcs, however. I probably phrased that poorly.

I don’t understand why secrets should be excluded from source control. It seems like a perfectly fine place for them to be stored to me especially when talking about secrets which the developers require in order to develop the code or maintain a set of systems.

If the secrets in source control are:

(1) encrypted

(2) never have decryption keys stored/loaded on any developer machine

(3) never have stored decrypted representations (only in memory representation of decrypted forms allowed when required).

If you follow these rules you won’t be more likely to accidentally commit unencrypted versions of the secrets and you’ll also by necessity have setup some (auditable) gatekeeper for logging of decryption events (via an aws kms or similar decryption as a service api).

For the category of secrets which must never exist in any decrypted form on a developer machine — maybe I can see the argument that those should be left out of source control as this would represent a reduction in the surface area for offline attacks against the encrypted form of the secrets -— but I would guess that this actually represents a somewhat minor gain in practice? And also seems strictly unimportant to use cases where access to the decrypted representation of the secrets on the developers machine is mandatory ...

It's so easy to avoid having them in VCS though. Any secrets can be loaded from a configuration file placed in ~. Populated configs can be selectively distributed so you can e.g. only place the production config file (with the prod server AWS keys for example) on the CI server. Everyone else just gets a config file with non-prod secrets. This can help avoid a lot of mishaps.

The problem with config files outside of vcs is keeping them in sync, which leads to bugs, insecure sharing methods, and constant interruptions as developers need to bug each other to get the latest values.

Shameless plug, but check out EnvKey[1].

With 10-15 minutes of effort, it gets secrets out of git, out of email, out of Slack, etc. It manages encryption keys safely behind the scenes, protects development secrets just as strongly as production, keeps devs and servers automatically in sync, and greatly simplifies access control and key rotation.

It's not the only solution out there, but it's by far the easiest to setup and work with. In any case, use something that truly solves the problem! Don't settle for half-measures that end up spraying secrets all over third party accounts. This stuff is serious--even when it comes to so-called development secrets, the line is fuzzy.

1 - https://www.envkey.com

Shameless or not, this is the first time I’ve seen a hazmat[1] handler I understood by the end of the first page and the price is low enough I’m just signing up now to use it for all my personal projects. Thank you for sharing it here.

1 - encryption keys and secrets are “hazardous material”, shortened to just “hazmat”. While necessary and arguably crucial in our work, they deserve the same care and respect a chemist would have for a beaker full of particularly dangerous chemicals.

Glad to hear it! Feel free to reach out with any feedback/questions/issues: dane [at] envkey.com

Let's assume that all the above is true.

Even then, secrets don't even belong in source control. They aren't tied to a specific version of your software -- they're tied to state in other systems: AWS, your MySQL database, a third-party API, etc. Those systems will change independently of any particular release of your code, and having it versioned in source control doesn't actually make sense.

Secrets belong wherever it makes the most sense for them to be, balancing operational needs, security, and compliance requirements.

Storing secrets in version control (with permissions to the repo tightly controlled), encrypted (with the decryption key only available on the compute needing secret access), and used for rendering into a discovery mechanism (or directly injected into instance env vars) is entirely legitimate (and I know of several orgs doing this).

They are tied to the overall state of a application though. Much like db versions, 3rd party apis etc.

You must track them somehow and a VCS is obviously the right thing to store them in.

Depending on the circumstances, I use git-encrypt, Ansible vault, vim -x (encrypt files) or even RCS (no chance of pushing to a repo when RCS doesn't have that feature).

This one of my criticisms of the 12 factor app. They say put the config in the environment. That doesn't address the question. How and why does it get into the environment and from where?

https://12factor.net/

Also, it really depends on the environment, the nature of secrets, etc. It's never great, but the level of badness varies dramatically.

If you have a team of 2-3 people, where everyone has access to everything anyway, and the secrets can't be used from outside the company (i.e. you need access to servers in the first place to authenticate with whatever service is configured), then it's one thing. It's bad, but it sure beats 'let's really carefully update it on all machines manually'.

I just really, really don't like blanket statements such as 'you shouldn't do that, full stop, context is irrelevant'.

Reply on news.ycombinator.com