Hacker News Reader

Comment by sushshshsh | Hacker News Reader

This is a good post because it details to me how horribly broken Shopify's process and culture are. Here's why:

If your software company revolves around getting a director or VP to say "you are allowed to do x technical thing", then your company has already lost. Here's what you need to do instead so you don't waste developer salaries and be outcompeted by better companies:

Every dev is allowed to spin up their own service. Teams can self organize and self govern around business problems they feel like solving. The best teams are the ones that produce the services that product owners rely on the most to get things done.

If you have a culture that involves saying no to dev ideas by default, it's only a matter of time until you lose.

View on HN

You can find gripes about process at every single company. Yet companies like Shopify are some of the biggest success stories and I know engineers who are very, very happy there.

You can give every engineer and team complete freedom, and you’ll end up with too many microservices to count, with ineffective higher level strategic alignment. The result? Horrible environment to get the things that matters done. Little upside for your stock.

Stop thinking of VPs and executives as harmful gatekeepers. Done right, they add incredible value to the business, and incredible value to your output as an employee. Done wrong though, yes, find another company; just as you should if engineering is dysfunctional.

If my company’s executives threw up their hands, and decided to stop doing their jobs and tell every team to solve whatever, I will look for another job immediately.

Every place has issues.

The top down these three people decide everything can lead to abuse or neglect. Most importantly it makes people do irrational personal things to win over the top three. This leads to internal cults.

Not necessarily true either. All organizations ideally have a goal. Vision. Mission, whatever you choose to call it. People at the top decide on that goal, and they decide on a strategy that might let them reach their goal.

That strategy should start by considering the things that cannot be changed (legislation like GDPR, maybe scalability, availability of resource, or health and safety regulations).

The leaders would then consider mutable factors, things they DO control like number of employees, technology options, business model, geographic location.

These two broad things (immutable and mutable factors) will lead to a list of things that need doing to achieve world domination. Sorry, I mean success.

That list of tasks is handed off to employees who will repeat a similar process within their areas at a more granular level.

Good reporting up and down the chain allows people at all levels of the organization to adjust course when needed, with a level of autonomy suited to whatever level you're in within the organization.

If the organisation is small enough people at the bottom may will seek out people at the top for direction.

No cult required.

"That list of tasks is handed off to employees who will repeat a similar process within their areas at a more granular level.

Good reporting up and down the chain allows people at all levels of the organization to adjust course when needed, with a level of autonomy suited to whatever level you're in within the organization."

Yeah this is the part that is missing at shopify from my experience. And it has caused all sorts of brutal problems and mental health issues for people.

Giving devs the green light to go crazy is also a recipe for disaster. Devs don't necessarily understand everything going on within a organization or even have a grasp on the security controls in place. In the last six months, I have experienced devs publishing secrets to open buckets, using insecure algorithms for creating customer and session ids, go to production with 3rd party vulnerable code, attempt to bypass a waf, attempt to bypass penetration testing and the list goes on. All of these things were done by developers with at least 4 years of "experience". Saying NO is sometimes the best way to save your business and brand from ending up on krebs as the latest breach.

Agreed. It's definitely a two-sided coin

Not saying that devs shouldn't be allowed to experiment, but a lot of times they need supervision.

By all means, check everything I boot up in AWS, but don’t prevent me from doing so in the first place.

My current company does this and it’s great. If I do something bad it gets destroyed in 30 minutes unless I fix it (we get notifications).

> Every dev is allowed to spin up their own service. Teams can self organize and self govern around business problems they feel like solving. The best teams are the ones that produce the services that product owners rely on the most to get things done.

Can't emphasize this enough. The only way to tackle complexity in tech organizations is by introducing at least one degree of freedom for innovation per employee.

Where would Uber be without three thousand microservices for three thousand engineers?

Making a profit.

Says more about the ridiculous profit margins than the software though.

I enjoy reading the spectrum of views on Uber, which seem to range from “they’re dumping rides below the cost of providing them” to “they have ridiculous profit margins”.

And yet a tons of SF companies never make a dime.

Tells a lot about the system right?

I recently discovered New Relic has yet to make any profit.

for the more credulous readers in the room I want to point out that the above comment is dripping with sarcasm. took me a while to determine it as well. poe's law is a cruel master

This sounds a lot like the Google approach, and I've seen a number of persuasive arguments that this is why they have 14 different, semi-competing chat apps and shutter most of their new services a couple years after introducing them. I expect somewhere in the middle there's an optimal balance between a completely top-down approach and one with too little hierarchy.

Of course the counter-argument is that Google still does quite well, but how much of that is due to their search and advertising monopoly, as opposed to the result of more recent innovation?

I’m sorry but this sounds like a nightmare where you end up in Uber-esque situations where 80% of engineers are reinventing every wheel with 1,000+ “microservices”. Nothing wrong with a hierarchical organization where everything is run like a tight ship.

What you’ve described is a recipe for endless technical debt and a culture that focuses on playing with whatever new tool is popular on HN this week instead of growing revenue or improving profit margins.

Have you considered handover, people moving on, new hires, etc? One guy running one service sounds like entrenchment to me. You need, as a company, to ensure people are replaceable. One way to do that is to for example stick to one or two programming languages.

You cannot, as a company, afford to have your services be written in a dozen different languages by a dozen different people. That's a huge risk.

It’s a huge risk if the unit of ownership is “engineer”. It’s less of a risk, but still not zero risk, if the unit of ownership is a squad/team/whatchamacallit, because you’ll get more brains applied to “should we really use Haskell for this?” and are isolated from the one Haskell engineer quitting. (Teams do rarely quit together, but individuals leave all the time in comparison.)

It’s also a risk to force your entire company to use 1-2 languages (one of which is [practically for most] hardwired to be Javascript, so you get to pick the other one).

At some points in time, c, c++, php, perl, or C# would have been reasonable choices for web dev. Do you want to be leading a company of Perl experts maintaining a large Perl codebase right now? Doing it, I can say that even C# is painful when it’s your mono-language monolith. (Inertia in language cuts both ways.)

Replies to your comment are unanimous so far:

"You can give every engineer and team complete freedom, and you’ll end up with too many microservices to count"

"Giving devs the green light to go crazy is also a recipe for disaster"

"Where would Uber be without three thousand microservices for three thousand engineers?"

"What you’ve described is a recipe for endless technical debt"

It's crazy that these replies do not even entertain the possibility that devs might be responsible enough to not abuse that freedom. I could spin up a hundred services tonight without approval but I obviously don't because it would be a nightmare to maintain, and I'm pretty sure my colleagues feel the same.

I'm getting a few downvotes, so let me try a different approach.

To the people I quoted: you evidently understand that runaway tech debt is a bad idea. So let's start a company where all of you are founders! At inception, this company would not need heavy approval processes around spinning new services and so on, since all of you share the same ideas about tech debt and can trust each other to do the right thing there. Security is more tricky, but all of you are responsible enough to actively seek feedback from security experts instead of just pushing insecure services to production, even though you would technically have the freedom to do that.

Now let's hire a few people to scale the company to 10 employees, while maintaining that same culture of responsibility. Now let's scale to 100. Then to 1000.

Why does necessarily stop working? Why is there necessarily a point where you cannot trust your colleagues to behave responsibly anymore?

(Note: this doesn't apply to everything. Personal information for example may legally require strict processes to be in place from day 1, and even if not legally required you may still want to do it by abundance of caution and respect for your users.)

Maybe all of the replies are from managers and CIOs :)

I agree, it keeps working just fine. You see it every day on the real internet.

You expose an API to me, I build a website around it. If one day your site disappears, I need to alreadt have backup strats in place on how I can get data like yours elsewhere. I have nobody to blame but myself-- after all, I'm getting the data from you for free!

You cannot maintain that same level of responsibility when you have 1000s of employees.

Depends if we're talking prototyping or production. Prototyping? Sure, go for it. Production? It depends. If you're building a social media platform for cats, knock yourself out. If you're in a regulated industry, e.g. financial, it will create headaches when the auditors and regulators come around.

The "where would Uber be" comment you quoted from seems to be expressing the opposite sentiment from the other two, at least the way I read it. Sure when I saw that thousand microservices talk I thought "what a (comparative) shit show" but many people admired the concept.

They decide what to work on? How does that work. Hey Joe, we need the payments module improved. Can you take a look? Nah, I’m rewriting the flux capacitor in JavaScript. Oh, ok. I didn’t know we had a flux capacitor. I’ll ask someone else.

Engineers are great - i am one - but the focus needs to come from the business.

If every dev is allowed to spin up their own service, what happens when the dev leaves? Or the service goes down while the dev is on vacation? What if the service becomes a single point of failure other devs' services rely on?

Thanks for the comment, let me try to answer :)

>if the dev leaves? Their code should be in a viewable repo and it should maybe even be documented. If it's not, no sweat, because I've seen lots of undocumented code in my life from bigger teams. Continue using the service until someone becomes an expert in it or the business problem is solved again. But, ideally you would have a few devs as experts on a few different services. It doesn't need to be 1:1.

>Service goes down while dev on vacation I'm thinking a production service should aim to have 2 devs and 2 ops supporting it. They could each be responsible for 2-3 major services and 2-3 minor or experimental services.

>Single point of failure Let's say you have a service that, if you give it a user id, returns only the history of that users orders. It doesn't fulfill orders or give delivery statuses or help initiate a customer support conversation. It is conceivable that if this service goes down in production, there may not be a way to get order history from anywhere else quickly. The information definitely exists in another db in another service somewhere, because that's the db where the order history service gets its information in a read only status. But it's not all neatly organized and available over HTTP. What do you do? You make the order history team own their SLA, otherwise their SLA will own them. In microservices world, you can't blame other people if you go down. You just have to do everything you can to keep your service running, by configuring AWS yourself and being on call yourself.

Hopefully at that point it’s not one devs service any more.

"...it's only a matter of time until you lose"

(unless your business has strong network effects)

Reply on news.ycombinator.com