Lessons from six months at Shopify
alexdanco.com
alexdanco.com
Reading this gave me the exact opposite impression - the amount of work it takes to extract takeaways from this writing is IMO one of it's biggest failures. It felt like pulling teeth to try and even get meaning out of some sections. #3 in particular is so nebulous I struggle to find the lesson at all. To me, this seems like a case of bad writing organization. If done better, I bet this piece would be half the length of current.
As to the contentious content: taking time to understand how your company works, great! Needing 6 months before being considered useful? Huge red flag.
> When you’re in a company full of smart people, like Shopify, it can often be quite tricky to resolve disagreements and impasses
Here's another huge red flag for me. The later explanation around having an idea of how the leadership/company thinks? Great. But having that philosophy/approach outlined should make resolution a lot easier. The "full of smart people" excuse could be rooted in a few things, but I struggle to think of a good one.
I don’t think this is ‘useful’ so much as ‘having a clue what is going on’.
It’s easy to be productive on the one project you are assigned to, but you won’t have any context for the implications of what you are working on.
> "In your first six months, you’re gonna be useless anyways. You’re going to be drowning in new information and context and it’ll take you a few months to learn how to swim."
Useless is their word here, and the way they continue to describe it gives the same impression.
I'm in agreement that knowing how organizations work takes time and experience, but six months is indicative of a larger problem in the org. Brevity and succinct, digestive philosophy is valuable. If you can't at least communicate a good outline to new hires in a month or two, something has gone wrong. The minutiae of course will take far longer than 6 months in total, but you should be able to "swim" by then, just not as fast as a 2-4 year vet.
- It really is true that big companies move much more slowly than startups.
- Consequently, startups should be wary of projects that depend on partnerships with big companies.
- Big companies are slow because knowledge is widely distributed and tough to wrangle, and liabilities and gotchas are numerous. Even simple things require a lot of know-how.
- To minimize slowness, find a single person who knows all of the moving parts involved. Avoid hodgepodges of people who each know only a tiny part of the whole challenge.
- It's important to start with this person, because problems snowball. An ounce of prevention is worth a pound of cure.
The storytelling flow of consciousness might work for some, but generally it's well established that for this type of writing, leading with your point and backing up with examples after tends to be more effective. But also in this case, a lot of the details don't actually add context to the lesson and can be removed without hurting the content of the piece.
As an example, here's a possible rewrite of section #3 that I think reads a lot better and really doesn't lack any material:
> A long time ago, I was told “if you’re a startup, try not to waste your time talking to big companies unless you’re doing it deliberately. They have an absolutely endless capacity to consume your time with meetings.” I didn't truly understand until I got a chance to work with a partner for my first time within Shopify on our partnership with Operation HOPE to help start 1,000,000 new Black-owned businesses over the next decade.
> It’s clear to me now how this happens. In a partnership, I have found the most important thing is making sure that someone who understands every piece of the project is involved from the very outset of the project. If this fails, you end up spending a lot of time (like I have) straightening out requirements and capabilities on both sides. This creates meeting after meeting as new groups get pulled in, but never solves the real problem.
Word Count: 488 vs 154
True. But this is why Goliaths get run over by Davids. When it take half a year's salary to begin getting return, that's six months of high risk. And then when your product cycles are six to twelve months, you're slow and venerable.
Full disclosure: I do some Shopify development. It's a love hate relationship. There are things that are great. But there are things where I stop and think "For Ford's sake??!!?? Really??" This article explains the excess of the latter.
Blockbuster... Could have invested in their David but didn't, I'm sure they're still regretting that.
To name a couple fairly obvious and well known examples of large incumbents being knocked down.
Amazon ate Walmart alive, alone with a handful of other retailers.
Etc.
Uber is global, your local tax company is not. Plus your local tax company is restrained by you know, laws.
Amazon vs Walmart is also not a good comparison .. I believe this david vs goliah is mostly start up people that think too highly of themselves! In reality you have Google, Facebook, Amazon, Apple, Microsoft... all huge companies that have been dominant for decade + and aren't too different from the post.
Walmart is actually doing very well, especially in e-commerce.
As it stands, you have the poorest Americans (the long tail) competing against a small section of the rise who can afford and trust that shipping goods to their door. The overlap is minor, excepting for perishables.
David is so far ahead Walmart is looking in the mirror thinking they're ahead :)
edit: The worst companies are the ones who claim they don't have office politics (ie: we're a flat meritocracy) but instead simply make it hidden and much harder to understand.
Correction: Office politics is how you work toward some goals in spite of other departments having other goals, some goals are incompatible/opposite and cause the other department to actively work against yours.
This sounds like a pure and narrow-minded engineering-like only POV. In reality, specially in big companies, you need a mix of both Politics and shipping great product. If you believe people make rational decisions all the time, you will be surprised with the amount of feelings that goes into making them. This post is great advice to anyone that wants to be a high performer anywhere in any field of work.
Most mentees accept it, but a growing minority are preemptively cynical about these things, dismissing them as "office politics". Especially for junior engineers who get a lot of their information from angry comment sections on HN and Reddit, there's a growing desire to view engineering as the core of the company and expect all other departments to serve engineering. This results in a lot of pain when they're dropped into a real company where they have to work side by side with other departments to generate revenue.
If you want to be successful, it's important to accept that communication, relationships, and rapport are just as critical to getting anything done as the engineering work itself. It's not office politics, it's just the nature of working with other people. The longer people resist this idea, the more time they waste fighting reality instead of finding a way to make things work.
It really doesn't take much effort to build relationships and rapport within a company. You're going to have a long, painful uphill battle if you spend all of your time resenting and fighting it, though.
The advice is probably good for any large organization, having worked at very successful startups with a few hundred people and mega-corps ... you absolutely have to understand how the decision making works if you want to get anything done.
FYI 'politics' is not bad, it's just all the bits that are unspoken about the operating nature of a company.
That is literally the one thing which I wish I had understood when venturing into a much bigger company.
In fact, what he should really say is 'I wish Shopify would formalize' the nature of how decisions are made etc..
Edit: I think younger people with less familiarity about the 'real nature' of large organizations might possibly misread that because it's sometimes hard to understand how some companies work, and it's easy to be cynical (frankly, many companies are 'political' in a bad way). In reality - this 'advice' is the revelation that anyone in middle management makes when they get into their 30's. Everything is making sausage, nothing is obvious except in hindsight, getting anything done is kind of hard, the company has limited ability to innovate on the margins, and social organizations are made of people, which means 'it's political' that doesn't mean it's bad, it just means you have to think about it differently. Middle management is hard, especially for those in staff positions without 'direct power' and are not high enough up to have meaningful legitimate authority for the scope of a project.
Much better to understand the working reality and either live with it or try to change it from a position of understanding.
This is what culture and history teach, whether its Xenophon or Succession.
Slightly off-topic generalisation, but maybe one of the reasons hackers disdain good management consultants, is that the latter are experts at understanding "the bits that are unspoken about the operating nature of a company", whereas the former see this as unnecessary overhead to doing what they want to do.
The people that are great are the ones that do both great work and can communicate.
It's probably one of those companies where fixing a bug a week is "productive" because everything is so hard to do. Ripe ground for the rest and vest types to hang around being dead weigh
Shopify main app is a giant Rails monolith. They also have several hundred other apps as well (eg, App Store, partners, burst).
Shopify has done a good job on defining clear interface boundaries between various parts of the monolith, which makes changes more compartmentalized. As well, they’ve invested significantly in developer tooling to make developers more efficient. It’s not perfect and there’s always more to do, but ask any Shopify developer about the magic of ‘dev up’ or about ShipIt.
I’m not quite sure how or why you extrapolate that having a large monolith immediately equates to having a culture cantered around low productivity. Perhaps you’re projecting from your own experience?
I’m still waiting for a AAA title that requires inferential analysis of financial statements. After all, as Ben Affleck demonstrates in The Accountant, there’s nothing quite so satisfying as explaining discrepancies in a receivables footnote via a thorough analysis of quarterly EBITDA reporting. The closest approach I’ve experienced in a game, both cognitively and emotionally speaking, is probably completing the Challenge in “The Witness”. I live in hope.
†(sorry)
In my opinion it is not a game, though. Eve:Online is a job.
If your software company revolves around getting a director or VP to say "you are allowed to do x technical thing", then your company has already lost. Here's what you need to do instead so you don't waste developer salaries and be outcompeted by better companies:
Every dev is allowed to spin up their own service. Teams can self organize and self govern around business problems they feel like solving. The best teams are the ones that produce the services that product owners rely on the most to get things done.
If you have a culture that involves saying no to dev ideas by default, it's only a matter of time until you lose.
You can give every engineer and team complete freedom, and you’ll end up with too many microservices to count, with ineffective higher level strategic alignment. The result? Horrible environment to get the things that matters done. Little upside for your stock.
Stop thinking of VPs and executives as harmful gatekeepers. Done right, they add incredible value to the business, and incredible value to your output as an employee. Done wrong though, yes, find another company; just as you should if engineering is dysfunctional.
If my company’s executives threw up their hands, and decided to stop doing their jobs and tell every team to solve whatever, I will look for another job immediately.
The top down these three people decide everything can lead to abuse or neglect. Most importantly it makes people do irrational personal things to win over the top three. This leads to internal cults.
That strategy should start by considering the things that cannot be changed (legislation like GDPR, maybe scalability, availability of resource, or health and safety regulations).
The leaders would then consider mutable factors, things they DO control like number of employees, technology options, business model, geographic location.
These two broad things (immutable and mutable factors) will lead to a list of things that need doing to achieve world domination. Sorry, I mean success.
That list of tasks is handed off to employees who will repeat a similar process within their areas at a more granular level.
Good reporting up and down the chain allows people at all levels of the organization to adjust course when needed, with a level of autonomy suited to whatever level you're in within the organization.
If the organisation is small enough people at the bottom may will seek out people at the top for direction.
No cult required.
Good reporting up and down the chain allows people at all levels of the organization to adjust course when needed, with a level of autonomy suited to whatever level you're in within the organization."
Yeah this is the part that is missing at shopify from my experience. And it has caused all sorts of brutal problems and mental health issues for people.
Not saying that devs shouldn't be allowed to experiment, but a lot of times they need supervision.
My current company does this and it’s great. If I do something bad it gets destroyed in 30 minutes unless I fix it (we get notifications).
Can't emphasize this enough. The only way to tackle complexity in tech organizations is by introducing at least one degree of freedom for innovation per employee.
Where would Uber be without three thousand microservices for three thousand engineers?
Tells a lot about the system right?
I recently discovered New Relic has yet to make any profit.
Of course the counter-argument is that Google still does quite well, but how much of that is due to their search and advertising monopoly, as opposed to the result of more recent innovation?
I’m sorry but this sounds like a nightmare where you end up in Uber-esque situations where 80% of engineers are reinventing every wheel with 1,000+ “microservices”. Nothing wrong with a hierarchical organization where everything is run like a tight ship.
Have you considered handover, people moving on, new hires, etc? One guy running one service sounds like entrenchment to me. You need, as a company, to ensure people are replaceable. One way to do that is to for example stick to one or two programming languages.
You cannot, as a company, afford to have your services be written in a dozen different languages by a dozen different people. That's a huge risk.
It’s also a risk to force your entire company to use 1-2 languages (one of which is [practically for most] hardwired to be Javascript, so you get to pick the other one).
At some points in time, c, c++, php, perl, or C# would have been reasonable choices for web dev. Do you want to be leading a company of Perl experts maintaining a large Perl codebase right now? Doing it, I can say that even C# is painful when it’s your mono-language monolith. (Inertia in language cuts both ways.)
"You can give every engineer and team complete freedom, and you’ll end up with too many microservices to count"
"Giving devs the green light to go crazy is also a recipe for disaster"
"Where would Uber be without three thousand microservices for three thousand engineers?"
"What you’ve described is a recipe for endless technical debt"
It's crazy that these replies do not even entertain the possibility that devs might be responsible enough to not abuse that freedom. I could spin up a hundred services tonight without approval but I obviously don't because it would be a nightmare to maintain, and I'm pretty sure my colleagues feel the same.
To the people I quoted: you evidently understand that runaway tech debt is a bad idea. So let's start a company where all of you are founders! At inception, this company would not need heavy approval processes around spinning new services and so on, since all of you share the same ideas about tech debt and can trust each other to do the right thing there. Security is more tricky, but all of you are responsible enough to actively seek feedback from security experts instead of just pushing insecure services to production, even though you would technically have the freedom to do that.
Now let's hire a few people to scale the company to 10 employees, while maintaining that same culture of responsibility. Now let's scale to 100. Then to 1000.
Why does necessarily stop working? Why is there necessarily a point where you cannot trust your colleagues to behave responsibly anymore?
(Note: this doesn't apply to everything. Personal information for example may legally require strict processes to be in place from day 1, and even if not legally required you may still want to do it by abundance of caution and respect for your users.)
I agree, it keeps working just fine. You see it every day on the real internet.
You expose an API to me, I build a website around it. If one day your site disappears, I need to alreadt have backup strats in place on how I can get data like yours elsewhere. I have nobody to blame but myself-- after all, I'm getting the data from you for free!
Engineers are great - i am one - but the focus needs to come from the business.
>if the dev leaves? Their code should be in a viewable repo and it should maybe even be documented. If it's not, no sweat, because I've seen lots of undocumented code in my life from bigger teams. Continue using the service until someone becomes an expert in it or the business problem is solved again. But, ideally you would have a few devs as experts on a few different services. It doesn't need to be 1:1.
>Service goes down while dev on vacation I'm thinking a production service should aim to have 2 devs and 2 ops supporting it. They could each be responsible for 2-3 major services and 2-3 minor or experimental services.
>Single point of failure Let's say you have a service that, if you give it a user id, returns only the history of that users orders. It doesn't fulfill orders or give delivery statuses or help initiate a customer support conversation. It is conceivable that if this service goes down in production, there may not be a way to get order history from anywhere else quickly. The information definitely exists in another db in another service somewhere, because that's the db where the order history service gets its information in a read only status. But it's not all neatly organized and available over HTTP. What do you do? You make the order history team own their SLA, otherwise their SLA will own them. In microservices world, you can't blame other people if you go down. You just have to do everything you can to keep your service running, by configuring AWS yourself and being on call yourself.
(unless your business has strong network effects)
I’ve worked at a place that had a similar sort of “artificial retention” due to a lack of local competition for talent, and curiously that place also had a very top-down and controlling executive team. It was still a great workplace but I was mystified on more than one occasion by how much time I had to spend getting permission from folks 2-3 layers above me in the organization.
Convincing people to listen and follow your ideas or deliver on your timeline can be very problems.
Also I can’t agree with point 5 enough. It really is true that the biggest (by volume at least) customers of software are software.
These are not lessons, these are random thoughts.
Shopify's website is starting to compete with Stripe for 'what am I even looking at'? I can always tell when sane engineers have completely lost control of a company and useless marketing and buzz-word-gurus have taken over. The cycle is complete when a company claims they're using 'AI' or 'machine learning' to help your business. Is Shopify there yet? I can't be bothered to click through their random offerings to find out.
kenrose: "I worked at Shopify in the past. Your assessment is incorrect. [...] Shopify has done a good job on defining clear interface boundaries between various parts of the monolith [...] they’ve invested significantly in developer tooling to make developers more efficient. It’s not perfect and there’s always more to do..."
sushshshsh: "This is a good post because it details to me how horribly broken Shopify's process and culture are."
In my work I'm exposed to many different online store and POS applications. Shopify is in a unique class. They have a diverse third-party apps community, and a documented API.
For my needs, their app community allowed me to extend Shopify reporting to suit my firm's particular requirements.
Other companies have their product and if their design doesn't fit your needs, is missing a feature in a report, or whatever, you're just SOL.
I think it's worth noting these details when reading about this person's experience. There might be a hint of it from the article: "Working on [Operation HOPE] was the first time I’ve ever really interacted with an outside organization from inside Shopify." (as near as I can tell this program is a drive to put more stores on the platform, and not including US based black owned _developers_ in their App Store).
Other comments here note the author does not identify their position (as developer), either.
Woefully inept people work there who suck up and miss managements ass because they don’t want to lose their perks.
Shopify is a mirage.
Translation? After 6 months with the company we'll find out if we should have offered you a position; and you'll find out if you should have accepted that offer?
Certainly, there's go to be a better way?
Also, going 6 months without feeling productive in some way, without feeling that you're contributing to the cause in some way feels like a big ask.
Again, is there not a better way?
Pardon the editorial but this sounds like a cult, not a 21st Century tech titan.
Turns out the team I'm working in (not above) really does not like change (i.e does not accept been-there-done-that-failed, as valid argument to do things different), looks down on 'business', often even on 'users' (technical pureness above everything else, above working software even) and most of all hates me for pointing out problems.
I'm still trying to find ways to fill my original 'problem' for which I came on board, but am also close to giving up, due to the negative energy all this is costing me.
So: good advice. But be certain the 'problem' you're hired for, is actually something that people want to solve.
Then I figured that just as I got rather vague direction on what I should do, there also wasn't anyone telling me what I should not do. So I started doing things in the intersection of being interesting and beneficial for the product.
So my advice would be: If there's no concrete problem to solve, find one that you like and be glad that you were given the freedom to do so.
But tbh I don't entirely see how it applies here
https://umaar.com/blog/lessons-learned-from-working-at-shaza...
Separately, I cannot possibly imagine an internal podcast is anything but awful.
I worked at a place with an internal podcast and never had time to listen during work hours and doubted I would learn anything I didn't from trusted water cooler comrades.
I just get Heaven's Gate vibes with an updated medium, honestly.
For a large firm getting everyone into the same room for a meeting isn't possible any more - either because they're remote (especially so these days) or there just isn't a room large enough. So something like a podcast or a Teams meeting becomes the best way to communicate with everyone.
Last time they were talking about remodelling the office to be more ‘open’, and I asked if anyone had given any consideration to how easy it would be to focus there? Nobody had.
This stuff where you’re supposed to be useless and confused for 6 whole months... that doesn’t seem good to me. Everything in our industry moves too fast for that, for one thing.
I vomited a bit into my mouth.