Most companies cant afford to throw large amounts of money at a team who’s entire job is managing a Jenkins pipeline.
That’s a part time project for a single developer at smaller places.
It's probably a good idea to take vacation whenever the person who owns the Jenkins pipeline announces that they've handed in their notice. Otherwise you might be invited to the handover meeting.
You basically take the person who's been working on something for thousands of hours most likely and has probably written a lot of work arounds and shortcuts to get their job done and they now have a 1 hour zoom session to brain dump their entire existance onto the poor souls who have to pick up the pieces when they are gone.
After that hour, they basically sign off with: glhf
I don't agree with this at the 300 engineer level; Specifically that they can't afford it. At that point, one or two more engineers is 1-2% increase in spend on compensation, and if you can get someone to handle architecture and developer support (ideally two different people) and empower them to make useful changes, you can probably get the ball rolling.
Now, if you're talking about a 20-50 engineer shop, the math can be a fair bit different and there I do agree with you.
It seems more down to the evergreen issue of tech debt and fragile tech stacks being a hard-to-quantify cost and mostly hidden from those making the budget decisions. No budget for maintenance or architecture -- things are done in the cheapest most expedient way possible. Then it becomes harder to make future changes, so doing it the right way starts costing even more, and the cycle repeats.
> Welcome, Neo, to the real world.
Indeed. Gamedev was frustrating for this quite often.
Being to a FAANG company (I worked for the F and the G) gives one a view of colossal tech stacks, some wonderful achievements of technological thought, some well thought-out processes (and also downsides, all well-known). It's a great learning experience, but a chance to make any publicly visible dent are pretty small.
I've also been at small startups. I worked somewhere early in my career where I was employee #5 and stayed there until we were about 70 employees. It was exciting and I can say humbly that I touched essentially every feature and decision that happened during those ~3 years, which can be rewarding. I'm not saying I personally built the whole company, i had other good engineers around me, but you were involved in everything, so you touched everything and played at least a partial role in everything that was built. That's a great learning experience in its own right. But I found it very stressful to have so much expected of you. At these small companies you will always have more on your plate than even the most efficient engineer can accomplish. The backlog is endless, and solving even basic problems often required new systems or decisions because everything is new and no precedent exists for anything.
I've found my happy place at what i call a "mid-size" company. By my defintiion that is ~250 employees (+/- 100). I admit that the industry definition of mid-size is generally 500-1000 though. I really like this 200-300 range size. It is big enough that at least basic systems exist for everything already. There are teams for each product meaning that you aren't building everything and can focus on a certain area. But you also aren't pigeonholed in a narrow job description either. While you might have a sector of the app you manage, it can still be moderately broad giving you good variety. The team isn't so small that you are doing everything, but its also still small enough that if you accomplish impressive achievements that it still gets noticed. It is small enough that you know most people in the company by name but the team is still large enough you can always find someone with expertise in a weird thing you might want help with. By the time the company is this size, they aren't going to dissapear overnight like a 20 person startup might (although in tech anything is still possible). I'm happy where I am now, so no plans on leaving. But when I search again, I am going to be targeting companies of 100-400 employees again. I really think its the perfect size.
Everything was built in-house, and none of it held a candle to industry-standard tools. Every single thing: observability, containerization, CI/CD, code-review, source-control, etc was mostly written in PHP with seemingly no UI input and no documentation.
The economy of scale largely favors the startup, because they simply rent best-in-world tools hosted by another company that has hundreds of people dedicated to source control or IDE (or whatever) versus 20 engineers a FANG company would put on it.
In a ZIRP world using the tools of other startups makes sense but the stability of the FAANGs and their in-house tooling means they can weather much choppier economic conditions than the equivalent startup.
AWS: Superior to the cloud compute at the Fang I was at. Best-in-class API, and documentation, and support on-call. Tenancy issues are rare.
Docker: Run any application on any server with one command. Isolates all state from that application to one-directory. Allows spinning up test-environments with your whole stack (~6 applications or so if you're a small team, including DB) on a server in a few minutes with docker-compose.
Github: Solid UI design, lots of little niceties, adequate search.
Splunk: Best-in-class logging solution (the place I was at asked you right-click download 200megs of logs and grep them). Can be intergated with Jenkins
Jenkins: Can do almost anything well, easily with the right plugins. Acts as both CI/CD job system and scheduled job system with full premissioning, history for auditing, logs, supports dynamic workers easily with AWS.
Packer, artifactory, slack for integrations, Jira (which can represent complex workflows well)
------
How they work at a startup - You can setup jenkins in a couple hours if you know what you're doing. Thanks to docker, it's 1-command to get it running. I'd then have nightly backups of it (easy via AWS or a jenkins job). Integrate it with LDAP so that you can lock down certain functionality if necessary (e.g. deploys, private creds). Set most jobs to have reasonable timeouts, retries, and alert the devops team on failures (devops will own making sure known-failure-cases emit the proper message and failure-code). Connect jenkins to splunk, and you'll have reliability and time statistics on every CI job.
Github PRs plug into jenkins, which can auto-lint, auto-build docker containers, and such.
Splunk (or sumo): Get logs / metrics / traces into here and/or datadog.
These types of connections aren’t anything like when I was in a huge multinational and was close with a handful of people. (Not everyone wants to participate in the group activities and that’s fine too.) (Also, again, I have this luxury because I don’t have dependents. I know this calculus doesn’t work for everyone.)
This is not a small tech company. This is medium to large. If the company itself has over 1,000 employees - it's a large company. It's just not a giant company like any of the FAANG types.
/s
I’m not sure why you added a /s in there.
If Im in a 300 person company I'm probably one of 40-60 devs, and most of my peers know me and have met me. If I'm a senior dev, there's a good chance I'm on first name basis with the executive team, and they know what I work on and the impact I have. There's a lot better chance of me being able to demonstrate my value and avoid being let go if cuts are needed.
At a 10000 person company I'm basically guaranteed to just be an employee number next to a salary number in a spreadsheet, when it comes time for cuts.
> "we made this service, we need a pipeline that does this, exactly this way, and we need it.... by Friday, let me know if I need to escalate it."
Whenever people threatened to escalate stuff, I would always laugh and say "go for it", to be honest I can't remember the last time I worked on something that wasn't "escalated". Basically the escalated pipeline was e size and our bandwidth was 0.9e - so we never would catch up and if it wasn't escalated it basically didn't exist.