kronin

Db-per-service provides many benefits, albeit with some drawbacks. If you are going to adopt microservices, you've already had to accept most of the drawbacks. Trying to mitigate those drawbacks by consolidating will end up losing much of the benefits of going to a microservice topology. Sagas or compensating transactions are a way to answer this concern without the complexity of a transaction manager.

If each service owns it's datastore, each service can choose the type of datastore that best fits it's use-case.

Each service can be owned and operated independently. Patches, upgrades, schema migrations, etc.

Each services' datastore can be scaled, backed up, restored, encrypted, etc independently all depending on the requirements, scale needs, compliance concerns, etc of a given service.

A badly behaving service can (mostly) not impact other services, where if they all shared a single database a single service could theoretically starve other services.

Yes, the distributed transaction problem is a drawback, but that can be partially mitigated through better definition of each service's bounded context.

If an architect that hasn't coded in 5 years or more dictates technological decisions to you, when they are out of touch of current best practices, how do you bridge that gap?

Building codes change. Architects are required to design buildings according to the new codes. There is no such thing in software.

When I first started in industry, storing plaintext passwords in the database was what everyone did. We have moved on from that. I've worked on j2ee web apps, where the EJB tier absolutely had to be on a separate server or servers. Times have moved on from there. I've met very few "software architects" that have kept current without writing code at least 5-10% of their job.

How can you design something that needs to be implemented in code if you yourself can't write code?

If you can't produce in the trenches, what makes you qualified to dictate what those that are in the trenches are doing?

Interesting examples.

I have interviewed candidates who: * Have impressive products that they "designed and implemented. * Co-authored a book. * Founded and ran a {programming-language} user group.

In all 3 cases, the candidates came across very well when talking at the surface level. Digging deeper, they (in the first example) overqualified their contribution, and none of the 3 could write the simplest of code.

There is a reason why coding tests exist.

You handle the hypothetical by validating a new environment can be built and bootstrapped. Doing this on a regular basis, either by tearing down and rebuilding dev, or in a separate environment just for this purpose, is not a "nice to have". This same problem exists with monoliths with complicated dependencies, nothing new here.

Just like backups, if infrastructure as code isn't tested, it's worthless.

Trusting your private network is private turns that whole network into a candy store once a beach head in that network has been established.

Defense in depth exists for a reason.

The risk with having a reset mentality, "after all it's only dev" (edit: not intending to imply you said this), is that the opportunity to learn is lost. If they took the reset tack, then this occurred in production and they didn't have that option available, choosing the reset path would suddenly be seen in hindsight as a lost opportunity.

Another possibility is that they deploy feature branches in dev, so for a given service there are more versions running simultaneously.

I tell the candidates that I will happily be their Google/stack overflow. Some candidates ask questions along that line, some don't.

I agree that we don't code in a vacuum. What I'm trying to ascertain in the interview is how someone thinks, how they approach a problem and work through the solution. The more they communicate the better, even if part of that communication is "is there a length property on the String class?"

I note, but don't give it too much weight, if there are minor syntax errors. I ask candidates to walk through their code with an example input, especially the ones that have errors. If they catch their errors and fix them, I note that. If they skip over their errors because they assume the code does what they wanted it to, I note that.

Ultimately, can you reason through the problem, come up with a solution, and talk about the tradeoffs you made in your implementation. Any senior should be able to handle that without any preparation.

I worked at a FAANG where in several loop debriefs I was challenged on the complexity of my question. One of the challengers asks a chutes and ladders question (given a random chutes and ladders board, where you can choose between 1-6 for every move, what is the minimum number of moves to get to the end).

It doesn't take a hard technical question to figure out whether someone knows how to think and work through problems.

I agree about your unspoken assumption, but many interviewees ask. And the post I was responding to pointed to the Apache commons library, so I hazard to say it's not universally known.

A fail is not a red flag. A senior candidate not asking clarifying questions does not disqualify them (what I would call a red flag) but it is brought up in the debrief as a potential gap. I don't expect them to state this could be implemented using a single call to Apache commons. I call those candidates out in the debrief as being aware of the library, as a bonus. But not saying it isn't a negative.

> There is a way to weed out the liars - you hire them and then when it does not work out you fire them.

In my experience, firing someone takes multiple months. Months in which they are mutating the codebase (with review, but it's a time sink that shouldn't exist for a decent senior), influencing other engineers, etc.

Passing on a good candidate is cheaper in the long run than hiring a bad candidate. Or said another way, false negatives are preferred over false positives.

> You would be astonished how little people lie on their CV.

This has not been my experience. From something as white-lieish as "I designed and implemented" vs. "I was a peer on the team that did it" to flat out fabrications. You learn alot by simply asking "what was your role on the project" and asking follow-ups to dig into their contributions.

My interview covers coding, data structures, algorithms, system design and data modeling. I attempt to tailor to the candidate, but there needs to be some level of standardization to be able to justify to the recruiters why a candidate didn't proceed.

Just because a candidate fails or falls short in one area doesn't mean they are cut, but it's a pretty large red flag if a candidate purports to know something, both on their resume and in person, yet can't answer some seemingly basic questions. If you state on your resume that you architected and implemented a system end to end, yet can't whiteboard the result and explain bottlenecks or failure conditions, I absolutely will assume you stretched the truth a bit. And if the candidate feels the need to do that to get the job, what will it be like to work with them? Could you trust their estimates? Would you be able to back them if something they delivered fails and you trusted them as a senior to deliver?

I ask follow-ups. I give hints. Honestly I'm dismayed at the skillset of purported "seniors" and "architects", and that's interviewing for small companies up to one of the FAANGs.

Since you're attacking me, do you have any experience screening resumes? Phone screens? In-person interviews? Have you had the experience of a candidate that has apparently "done it all" on their resume, and confirmed their skill level when asked in-person, only to see them struggle with the simplest question in an area they deem themself an "expert" in?

I've interviewed well over 300 people, for positions from junior/entry-level to principal. If you read many of the comments to this article you'll hear other people with experience interviewing who also call out that candidates can and do lie. I don't like it, I'd rather it not be the case. I constrain the majority of my questioning to what a candidate claims to know on their resume. Here's a tip... If you don't know it (and I'm not talking about bullshit trivia questions), then don't put it on your resume.

I asked a candidate to implement an algorithm that reverses a string. I don't see that as a complicated problem. It's literally a for loop, and based on how they choose to implement it you can ask about implications of append vs. prepend, etc.

I didn't ask them to implement dijkstra's algorithm, or even something as "hard" as breadth-first or depth-first search (which I feel any senior that deals with trees should be able to do).

It's a marker of this... Can the candidate solve a basic problem? Once they solve it, can they explain their solution? Do they understand the memory/time tradeoffs they chose? Do they know whether the code they wrote is thread-safe? If you don't feel a senior engineer should be able to answer those questions, to a very basic problem, how do you expect them to tackle something more complex?

It's the start of the coding interview. It moves on from there. This candidate didn't even ask if he could use Apache commons, and I hadn't yet specified that he couldn't use a 3rd party library. So yes. He failed on many counts.

1) not asking clarifying questions 2) not being familiar with some well-known libraries 3) not being able to implement a basic reverse string 4) not being able to explain whether or not a simple method was thread safe (with multi-threading experience all over his resume)

I expect engineers, senior or otherwise, to be able to write code. If you can't do that, don't put it on your resume.

If you honestly think implementing a reverse string method is "too complicated"...

Edit: sorry, leaving the response as-is, but I recognize you didn't state "too complicated". You stated "focused on the wrong skills". I posit that an engineer that can't reverse a string, as a litmus test, will be likely unable to solve a more complicated problem.

I give them a chance to judge their own skillset. After looking at their resume, I have an idea based on the projects they take credit for, what their experience level should be. If what they state lines up with their resume, and they are unable to answer a simple question, with multiple hints and help, then I absolutely assume they are a liar (edit: or delusional, which is even worse. An expectation I have of seniors and above is to have some idea of knowing what they don't know).

It would be unreasonable for me to pass them through an interview that the other members on the team passed through based on "oh, they had a bad day".

Since you haven't been on the other side of the table, have you had coworkers who obviously were at a position above their skill level? Who allowed the rest of their team to carry them?

I've had periods wherey personal life has affected my productivity. That doesn't preclude my ability to write an algorithm that reverses a string nor write a simple SQL join. I don't ask brain teasers. I ask questions related to what the engineers under me will be expected to solve every day.

I've interviewed and been interviewing for over a decade. I have had the displeasure of interviewing many people who have very impressive resumes yet when asked how they would judge their skillset in an area, and being told from them that they would say "expert", being shown exactly the opposite when it came time to answer some questions in that area.

One data point... Senior engineer that founded the local java user group and wrote a published book on the java language. Couldn't code a simple reverse string algorithm, nor explain whether, after with help devising an answer, the method was thread-safe.

Another data point... A self-described expert in SQL with it all over their resume. Couldn't write a simple join, inner outer or other.

Weed-out questions exist because our industry is flooded with people who don't know, nor care to know, their craft. And there are plenty of employers who continue to hire these people, and promote them. I interviewed numerous "architects" (with development experience all over their resume) who couldn't come up with a naive solution, let alone the optimal, to a very simple problem. Want to fix the interview process in our industry? Figure out a way to weed out the liars? I'm all for it. I would rather not have to do basic algorithm questions, but that's the current environment.

No, I was answering the statement "it looks like an absolute nightmare to debug".

Amazon's page request and rendering pipeline, I would estimate, is necessary complexity. Due to the large engineering force and the speed at which they want to move, it's imperative that the infrastructure support distributed teams delivering at their own pace while minimizing integration points via shared repos but instead integrate via defined APIs. This is a fallout of that need. Do most companies have that need? Probably not.

Amazon has invested significantly in tooling that makes debugging this possible.

I would not recommend anyone go down this route unless they are also willing to invest heavily in the tooling support necessary.

The ports that are common are the envoy ports. In the roadmap it's mentioned that support for istio mixer is planned implying that mixer isn't supported now which, if it was istio, it would be.

Thanks for this. "Unlimited tolerance must lead to the disappearance of tolerance."

It seems to hinge on "rational argument" to base determination on whether tolerance is warranted, lest intolerance creep in. I wasn't able to discern the bar for "rational argument". Do you have some more info?

"tolerant of intolerance"... Interesting concept. Who defines the intolerance?

"tolerance" swings both ways. Unless you choose to be intolerant.

The majority of people who spout tolerance and acceptance at me are the least tolerant and acceptant of other beliefs.

Agreed re: both of these wins.

On the security front, would be ideal if these resulting containers ran as non-root. Hopefully that will come soon.

Spring boot microservices using liquibase as schema evolver, we are using helm hooks to run a job pre-deploy that leverages a spring boot profile which runs the evolutions. If it fails, return non-zero exit code and the deploy doesn't continue. The only pain point we've run into with this model is that the configmaps that the service depends on (we use the same docker image in both the deployment and evolution job) aren't created before the hook is called, so we need to duplicate part of the config with -D startup params.

1,000x this. A simple refactor commit, with a comment stating "reformat to match current coding standards. No functional changes".

Repos. Watch the Kelsey Hightower keynote video I added to my post.

Don't get me wrong, I think auto devops is a good thing, but it's most certainly not gitops.

Please don't.

I am using gitlab, though we quickly grew beyond auto devops.

Your definition of auto devops is different than gitops. Gitops is the practice of using commits and pull requests to execute change and do releases. Weave uses it to mean git as the source of truth. https://www.weave.works/blog/gitops-operations-by-pull-reque... Kelsey Hightower talked about it and and has demoed the workflow of using pull requests to initiate promotion and deployments.

Gitlab's auto devops does not seem to tackle promotion via environment repos, so in my understanding does not fit gitops and would be confusing to call it such.

Edit: Kelsey Hightower's kubeconf talk - https://www.youtube.com/watch?v=07jq-5VbBVQ

And a better writeup on weave's site -https://www.weave.works/blog/gitops-high-velocity-cicd-for-k...

Seems that metrics providing visibility into the "network connectivity was flaky", like looking at response times (particularly 95/99 percentile) and digging into the pod, which gives you the node, would have isolated the problem pretty quickly to a single node. If a problem is isolated to a node, first thing to look at would be node logs. Would that pattern not have worked in this case?

You can use IAM to auth to a cluster using heptio authenticator https://github.com/heptio/authenticator/blob/master/README.m...

You can grant IAM roles to individual pods running in k8s using kube2iam, though there are new advancements seemingly coming, or already out, now that amazon has announced eks. https://github.com/jtblin/kube2iam

Kops can provision and manage a cluster that incorporates multiple subnets allowing you to have a multi-az buildout https://github.com/kubernetes/kops/blob/master/docs/high_ava...