Screwdriver: Yahoo's Continuous Delivery Build System for Dynamic Infrastructure | Hacker News Reader

Screwdriver: Yahoo's Continuous Delivery Build System for Dynamic Infrastructure

yahooeng.tumblr.com

154 pointsphotonwins58 comments

Open articleView on HN

For the confused: CD = Continuous Delivery, not Compact Disc.

Mods: Perhaps edit the title?

Agreed, but I don't see an option to update/edit the title. Do I need to have more karma points for that?

No, you pretty much have to wait for the mods to notice it.

HN needs a better mechanism to highlight posts for the mods.

I think that mechanism is to write an email to hn@ycombinator.com

IMO let's not waste the mods time with this, I feel that 99.9% of the visitors would associate the term CD with continuous dilevery and not compact disc

I never even read "Continuous Delivery" before today, so no, that does not seem like the case.

This is genuinely interesting to me, see my comment here: https://news.ycombinator.com/item?id=13425660

I felt like the CD = continuous delivery was obvious. But perhaps I've spent to much time on Dev Ops. Maybe simply saying CI/CD would be sufficient.

Obvious to me too, but people doing CI, even a basic implementation like compiling, are still a minority of developers.

Not entirely obvious to me. I mean, we are talking about Yahoo! and I remember when I used to get those AOL CDs in the mail on a weekly basis.

When I read "CI/CD", I think "Corporate Identity / Corporate Design", which is just as confusing, if not more so... :-?

CI yes, CD no - in my mind CD is pretty much fixed to compact disc for eternity.

When I first read the title I thought "oh, Yahoo must have had some cool automatic system to burn CDs that is probably so old and pointless that they are opensourcing it now as part of their mothballing". I was fairly disappointed to see it's Yet Another CI Tool.

Haha, I had exactly this...

I confess I read it as Compact Disk. This made me curious enough to read the post, which I probably wouldn't have bothered to do otherwise. It was interesting though.

Are physical CDs still a thing?

No, hipsters are only just upgrading to tapes.

Heh, first thing that sprung to mind was what.tape, I hear it's the new big thing. Seriously though - there's no good reason for anyone to embrace standard cassette tapes, vinyl I understand - not because of any audio quality based arguments which are easily debunked but because the large format tactile experience of the physical and visual experience of the playing a large rotating medium with friends can be quite compelling. </endtangent>

More now that what.cd is gone.

They're still big in classical music. And, yes, classical music is still a thing too! ;-)

Jazz too, because of the older audience still buying physical goods.

Worship music as well (think churches).

I was kind of thinking the same thing, the last time I had a computer or laptop with an optical drive must have been something like 8+ years ago and even then I can't remember using it for years.

Honestly - I don't think most people would confuse this unless they're still on dialup

*Edit: sorry, my comment was intended as humour but was actually a bit negative

Sorry for chiming in late, had to wait for someone to get off the phone before I could dial in.

Joking aside, I did think CD was a Compact Disc, but that's probably because I grew up with CDs and still have them hanging around the house.

Then I thought it could be to do with changing directory for a split second.

Please don't be sorry! I'm genuinely surprised that several people experience a different association with the acronym with relation to technology and open sourcing 'things' especially in the HN space. I grew up with CDs, however as I mentioned in another comment I haven't used or had the means to use optical media in over a decade nor do I see optical media present at work or with friends so it's quite interesting to me, I'm not convinced that I'm in some sort of 'bubble' but more that perhaps that it could be something like a generational association. If you don't mind me asking, in the name of science - what age bracket do you fall under? I'm 30 or there abouts and for me I think optical media became something irrelevant or perhaps irreverent in my technical / cultural experiences in around the time of 2006-2008 and continuous delivery / integration became a valuable focus of product delivery and technical engineering at around 2010 and something that I've been experienced since around 2011ish.

I think you've hit something quite relevant here. While compact discs aren't something I deal with daily anymore, I've never been directly involved in continuous delivery. So I feel that it's because I haven't overwritten the acronym as of yet with a more relevant context.

I think it's just industry terminology for someone not in that exact space.

I'm 34 by the way.

I'm genuinely surprised about this too. 5 minutes ago I would have been certain CD = "Continuous Delivery" with any group of developers, even more so HN. It makes me wonder what other assumptions I'm making or things I'm taking for granted.

40ish. I still use DVDs almost every week: buying (mostly pre 2005) movies and rip them to my Plex for viewing.

I initially thought it was a story about some obscure compact disc pressing/inventory management system that Yahoo apparently had lying around because big corps have strange stuff in their basements. I mean, it's pretty common to see curiosity/historic articles on HN. 35 years of age ;-)

I just hope Digital Versatile Delivery doesn't become a popular phrase in the future.

Introducing: Design Validation & Delivery.

You're welcome. ;).

why would someone think it was "compact disc"? even if a hn reader was somehow completely ignorant of continuous delivery, the context and usage should be a guide.

I was gonna put up something full of sarcasm. Then I realize engineers are the least responsible for enterprise business failure. This could happen to any of us. Company goes under water and job security fades away. Thanks for contributing to the open source world. And best of luck to all of those brother and sister engineers.

Job security does not fade away for software developers for now

I think it's extremely unlikely that the higher ups who ran Yahoo into the ground had any idea of the talent they wasted. Nor do I think they would care then, or now even if they did. After all, apparently working from home was verboten at some stage during MM's reign...

Although Yahoo doesn't seem to be doing very well recently, they've done lots of great engineering. I recently looked through the organization's Github [0] and discovered a few nice packages.

I'd happily pay for a good managed CD build system capable of handling projects / products with multiple dependencies.

Right now I use CircleCI along with a bunch of scripts, and it works, but it's not great. One problem with existing CI tools that I've used is they only handle a single git repo, so extra automation has to be done ad-hoc. It's not terribly difficult, but it's tedious and it doesn't give you any kind of safety guarantees.

[0] https://github.com/yahoo

Fwiw https://gocd.io/ handles pipelines (or graphs of multiple piplines) with several git repos just fine. That's on-premise, but not hard to run on your own.

There's Snap CI, which is by ThoughtWorks (just as Go CD), and that's in the cloud. I don't know if they use the same code base, but it might be worth investigating if they can deal with multiple repos in a sensible way.

We just switched to GoCD. And here I thought Jenkins sucked.

I've never missed simple shell scripts, builds triggered by commits, tailing logs so much. Being closer to the metal. Back then, I had no idea the obvious solution was called "continuous integration".

GoCD has a pretty dash board though. So I guess that's progress.

>So I guess that's progress.

No it's not, because scripts are better to debug the problem when (invariably) all goes to shit.

You should try concourse.ci then. I tried Circle gitlab go and concourse and concourse is the best so far.

I wasn't responsible for setup, but we've been using Buildkite (https://buildkite.com) for our CI and it's been very nice. We run it through a Kubernetes cluster of workers and as far as I can tell it's been very flexible as to what's in a pipeline.

There are only 3 ways to model CI:

- Plain jobs (disconnected dots) (Jenkins/Travis-CI/etc)

- Full-on graphs with value stream maps and arbitrary connections (GoCD/Concourse/etc)

- Isolated pipelines (disconnected matrices/lists of columns) (Snap-CI/GitLab CI/etc)

I've lamented for a long time that there was no "lightweight" self-hosted pipeline tool that makes full use of modern tech (lightweight API, separate UI, offload the actual build environment to containers) and does nothing more than linking the pipeline and gets out of the way. Most organizations just don't have the engineering skills to handle build graphs. And the alternative job-style ends up in a complete mess.

Screwdriver fills a real void. I'm not a fanboy by any chance, but I'm really thankful to Yahoo for opening up this void so that many will follow.

Btw, Jenkins 2 Pipelines are not even in the picture because are neither proper pipelines nor proper jobs. It has no resumable/pausable stages, and is just a glorified DSL/UI on top of a multi-job kind of model.

> Btw, Jenkins 2 Pipelines are not even in the picture because are neither proper pipelines nor proper jobs. It has no resumable/pausable stages, and is just a glorified DSL/UI on top of a multi-job kind of model.

That glorified DSL is Apache Groovy. Its backers have been trying to distribute it by getting other JVM-based products to bundle it as a DSL, without worrying about how closely it really matches the product's functionality, such as specifying pause/resume in pipelines.

The screenshot looks a bit like Chef's Automate. The config example looks like Concourse's config (http://docs.screwdriver.cd/user-guide/configuration/index#ya...), but it doesn't look like they have task files.

They should write up a comparison to GoCD, Teamcity, Concourse, Jenkins, Chef Automate, etc. It can be pretty hard to evaluate these things without losing a month.

Chef cannot rollback. Chef is mainly for deployment but I think screwdriver focuses on CI. CD is not his business.

I mean, Chef Automate doesn't support rollback after all the usual toe dipping (https://docs.chef.io/workflow.html), but that isn't terrible. Automatic rollback generally comes with a fat "but not the database" asterisk, and if you are building something general, it's a pretty reasonable decision.

Screwdrivers docs all seem to mention CD, not sure why you'd say that.

I think in general it's becoming an intractable task to evaluate all the tools we have at our disposal.

In my case, I've been using CircleCI (mostly good, but a few issues make it a non-starter), CodeFresh (awesome, but very new, still needs work), and the new Jenkins. Wanted to look at Concourse and a couple others, but just don't have the time to test drive them in any real capacity.

Increasingly I think us software devs need to share our battle-won knowledge instead of just trusting the marketing blurb on a project page. Much of what I do these days just amounts to making guesses for suitable tools/projects based on heuristics and gut feelings instead of actual experience.

Hopefully something like StackShare will gain in popularity, and we can start sharing our experiences more.

Now time to check out Screwdriver!

Just spent the last 15minutes on it and my first impression is that documentation is still too poor to use it professionally. Even though, after a couple of tries, it looks like an awesome start and I'll follow this up with great attention.

Good job to the team for building an open-source tool for CD!

Anyone know of a tool that can create those nice diagrams?

Honestly the yahoo engineers are doing a fantastic job. First the NSFW open source model, now this, easy to use (got started in 3 minutes on my servers). A whole lotta love from Germany guys and gals <3

I just tried running the docker image from the readme, it doesnt exist? docker.io/screwdrivercd/api

Probably should just be https://hub.docker.com/r/screwdrivercd/screwdriver/ perhaps?

There are a few bits: https://hub.docker.com/u/screwdrivercd/

Oh, thanks! PR submitted: https://github.com/screwdriver-cd/screwdriver/pull/428

So if I'm already using gitlab or something similar is there a reason I should consider switching?

Edit: I also read this as compact disk at first.

"A peek under the purple rug!" sounds awfully suggestive. I hope that's as obvious to everyone else.

Purple rug = Yahoo internals

Is there anything I am missing?

I think that's just you, buddy. The only thing it suggests to me is the need for a more engaging tagline.

(But maybe it's just a matter of perspective. I once knew a redheaded woman who'd replaced a specific subset of her body hair with a brightly colored flame tattoo, which against the very pale skin typical of redheads produced an effect both immediately striking, and quite memorable at what is now a considerable distance of time. Had she instead dyed that hair purple, perhaps I'd be more likely to see this matter in the same way you seem to.)

Reply on news.ycombinator.com