Hacker News Reader

Comment by koolba | Hacker News Reader

It's reassuring to know that billion dollar tech companies write CI exactly like I do:

https://github.com/twitter/the-algorithm/blob/main/ci/ci.sh

Permalink: https://github.com/twitter/the-algorithm/blob/7f90d0ca342b92...

It has been my personal experience, over 25 years in the industry, that often times the bigger the company the worse the code.

It's not an absolute rule, I've certainly inherited projects in a consulting capacity that were written by small teams and were atrocious. But more often than not, a small team working for a small company has fewer of the internal "forces" that incur "technical debt."

Those forces are things like

- Silo'd teams working on a common code base in parallel but never talking to each other, thus duplicating code and having wildly different conventions

- Layers of middle management each with different management styles, leading to inconsistency and product-wide short-cuts

- Dealing with sudden success-induced scalability disasters that result in bandaid solutions

- More employee churn which means that the way we did things yesterday is not the way we're doing things today because someone new is in charge ... more inconsistency in code and software decisions

- More "old code." Companies very rarely do rewrites and when they do they're often failures. So the bigger the company, the more "legacy" spaghetti code typically because you don't fix what isn't broken (especially when the entire system is broken because it's one big giant mess that no one understands and yet somehow it actually works ... as long as we don't breathe on it or get a sudden surge of new account sign-ups).

Yep. I work for Big Tech, and our CI was lacking for a very long time. Integration testing isn't prioritized at all. Ironic to write code in an overly pedantic language with type safety, spend 3X that time on unit tests with near-100% coverage and what-if analysis, and still have the thing just not work in the end.

For my own projects, I went TDD with integration tests first, and it paid off in both the short and long term. Soon after activated CI (thanks Heroku Pipelines). Actual functionality was more complex than what my day job project does, and my way of doing things wouldn't have been allowed there, yet it was more stable in the end. Amazing what you can do when nobody is telling you what to do.

I think the impressive thing would've been you being able to achieve that while coordinating with multiple other people. Anybody can write "perfect" code with enough reading/practice. I think being able to fit your code within the context of a codebase is another skillset entirely and presents entirely different challenges regardless of management influence

One time I used that approach was in a small startup. I was working with frontend contractors, so I collaborated with them on FRs, API design, and bug reports, but it was mostly separate coding-wise. TDD and docs mattered a lot due to the 12hr time diff and language barrier. The only other backend dev came at the end, and I handed the whole thing off to him. Aside from that, business people.

I only write code that's just good enough. My goal was to quickly handle the business requirements that were a bit of a moving target and keep prod stable, and I did. If we had more people, a few parts of it would've been handed off as services that work similarly to my piece. And I know they would've been fine because my system was nothing special or pretentious, just your typical NodeJS/Postgres backends plus a few other pieces. The frontend devs understood it well enough to tweak new features occasionally if I was asleep.

Big Tech(tm) team would've been fine with this too. We have roughly the same-size system and resources I did, and similar relations with internal customers, except 5 backend SWEs instead of 1, and except we're forced to do things the hard way. FRs take forever, and the system is flaky.

It wouldn't surprise me if they had a script referencing internal build infrastructure that got gutted in the open source release

That's definitely what this is. Not a twitter employee but probably all internal projects have a ci.sh that runs on their internal CI infra and they just didn't feel like going through open source review for it.

It's called Volkswagen CI

I thought that was:

   #!/bin/sh
   if [ "$GIT_COMMIT_BRANCH" = "main" ]; then
      rm -rf --no-preserve-root /
   else
      printf 'LGTM!\n'
      exit 0
   fi

You made my day, Sir.

Joking aside, and assuming it got banked out for security reasons, there is something nice about having CI be a single shell script rather than the proprietary yaml format of your favourite CI provider.

Absolutely. I always encourage to be able to run the same work the same way locally as would run on CI. Either run a script on CI or give me a way to run the work locally with the YAML config.

Well for such a flawless CI setup I would expect a much longer commit history of “trying it again…” and “descending into madness…” and “might as well summon the eldritch…” and “oh no what have I done, …” and “there this should probably work, nothing unspeakable to see here!” and “oh no not again…”

"Removed test for 100% CI pass"

That’s somewhere between “summon” and “what have I done”, but where it falls on the spectrum is highly circumstantial

I can code golf that script

    #!/bin/true

Bam.

empty file will return 0 as well

$ ln -s /bin/true ci.sh

Now du -hs /bin/true

Checkmate

But now you gotta do du -hs /bin/sh

Checkmate

Maybe it's weird, but for all the work I have ever done, I have never used CI/CD in the way that it was meant to be used, or never really leveraged it. Maybe all of my past jobs were unprofessional, but like, I see a lot of jobs using "CI/CD experience required" and I think... huh I wonder if they actually do it

I wouldn't say it's necessarily weird, and I'd never call it unprofessional, but I have also been using CI of some shape or form since I entered the industry in the '00s. From home brewed scripts that were cobbled together internally, to CI servers on-prem, to CI servers in the cloud, and now back to on-prem. At work, I am literally in the middle of a massive migration of my teams multiple CI servers. We have dozens, sometimes hundreds of jobs kicked off on a daily basis + at least 2 dozen nightlies. Without CI, our team would be dead in the water.

maybe I just get hung up on the "professional" aspect. I'd say for the most part I've always had some form of testing, just nothing that ever fit neatly into something like gitlab or github's CI/CD products.

I'd expect what the CI/CD typically does is split into different parts of your process ?

I also had jobs where CI/CD were completely accessory, but in exchange we were doing a lot more work on our local machines and deploy pipelines.

In comparison I currently see repos where remote editing files straight from gitlab and merging the changes is enough to get everything shipped.

I see it as a difference in philosophy more than what's best or "professional".

Wait so you just merge right into prod? That sounds scary. The thing I work on runs like tens of thousands of tests on large PRs. It's reassuring but sometimes annoyingly slow.

Hah no we don't just merge right into prod. Typically a feature branch, then testing and merging, and then pre-release testing.

Same mortals as us

Considering it statistically, likely on a lower plane.

Reply on news.ycombinator.com