Hacker News Reader

> I use Rake (or Gulp, or whatever) because then I can use Ruby (or JavaScript, or whatever).

You can use Ruby in Make. I don't know Ruby but here's some Python:

  SHELL = python
  .SHELLFLAGS = -c
  .ONESHELL:
  .DELETE_ON_ERROR:

  foo.txt: bar.json
      import json
      with open('$<') as input, open('$@', 'w') as output:
        output.write(json.load(input)['the_text'])

(Caveat: I've never actually tried this at scale.)

You can even use different languages for different recipes with per-recipe variable settings.

Ha, that's a neat trick! Trouble is, for either Python or Ruby it becomes tricky due to stuff like dependency management. You'll have to `bundle exec make` to get sane library paths for Ruby or `pipenv run` for Python, etcetera etcetera.

At that point I think you might as well just use a language-native one.

Great pull, though.

The GP does this via a neat hack, but you can also do this in a much more understandable fashion by simply having the body of every Makefile rule start out by shelling out to some script in your favorite language.

I think you're (understandably) misinformed about what Makefiles do because you've run into some bad ones. The thing they're doing is managing a N-level deep dependency tree in a declarative way. So if A->B->C you can run something to generate C, then B can run, and finally A, and this can all be done in parallel for hundreds of files.

On the individual rule level this is really simple, e.g. just turning a .c file into a .o file, then finally some rule that depends on all *.o being generated creates a program out of them.

The language-native ones are usually much worse. They're easier to use at the outset because they don't make you create this dependency graph, but that also means that they can't run in parallel on your N cores, and they usually suck at incrementally updating the build.

> The GP does this via a neat hack, but you can also do this in a much more understandable fashion by simply having the body of every Makefile rule start out by shelling out to some script in your favorite language.

I'm not sure what you mean? How would this allow you to use, say, Python as the language for recipes? Just having Make drop straight into Python kind of defeats the purpose of Make.

You'd use Python as the language for the recipe that turns (in this example) a given .c file into a .o file, while leaving the Makefile to do what it's good at, declaring the DAG dependency tree needed to incrementally build it.

The point is that people conflate these two things. They open some random Makefile and see that it's mostly doing stuff with shellscripts, and think "oh I should do this all in Python", and then write some monstrosity that doesn't make a dependency DAG and thus can't run in parallel or be easily understood.

Instead they should have split the actual logic they found in shellscripts in to Python scripts the Makefile invokes.

Nevermind, I misread you. I missed "rule" where you said "beginning of every Makefile rule". (I thought you were suggesting just having the default rule run some enormous Python script, which I've unfortunately seen before.)

> Shell is the lowest common denominator. Expecting everything to at all times be written in and for that lowest common denominator is not reasonable

Is it that hard to learn shell? Why is it so painful? What makes it the "lowest common denominator"? I use it all the time, but I admit at work I am one of the few.

> Expecting everything to at all times be written in and for that lowest common denominator is not reasonable. We're a tool-using species and we refine tools over time to make them better.

This is too vague. What makes a Ruby based build or JS based build a more "refined" tool? It sounds like familiarity is the real issue here.

> If old tools were sufficient, people would use them because learning new ones is hard.

How many people even know Makefiles these days anyway? The "modern" approach seems to be, learn a programming language and then try to do everything inside of it. Some languages are more interested in this cloistered philosophy than others (like JS).

If anything, I think the reason these build tools keep being proliferated is because nobody wants to learn anything more than the bare minimum to "be productive" (which, depending on what you're working on, can be anything from pushing out customer demos for a company that will never sell, to microservices operating at scale). Learning a language and never leaving its paradigms/comforts is easy.

Your second paragraph got to the heart of it. If we want to use some standard build toolchain, it needs to use a nice language and not feel obscure. I was explaining to someone a bash script I wrote, and he said "why not use Python". There were reasons but... he was right, Python would be much easier to use and maintain, and we have a lot more developers who know it.

That said, Maven is incredibly suck-tastic.

Eh. It's not my favorite thing out there, but Maven's fine for what it is. It's designed for and explicitly for well-behaved Java artifacts. If your Java artifacts are not well-behaved, you're going to have a bad time--in my experience, most of those cases are doing things you probably shouldn't be doing.

(You may be a wizard and have a reason to do them, for sure--but that's what writing Maven plugins is for. Or not using Maven. You've got choices.)

Given the limitations of the platform, there really isn't a such a thing as a well-behaved JVM library that depends on other libraries, unfortunately. Oracle really dropped the ball by only serving their own needs with the module system.

Can you expand on this? Having done a pretty decent bit of JVM development, I've never really run into issues even doing some not-out-of-the-box stuff.

What reasons? I did some ruby shell automation and it’s dead simple to get it working and working correctly. I imagine python has a similar story.

> `JSON.parse` and a working subscript operator

https://github.com/ingydotnet/json-bash

    source json.bash
    
    json='{"name":"Jason","friends":["Jimmy","Joe"]}'
    JSON.load "$json"
    joe=$(JSON.get /friends/1)
    JSON.put /friends/2 Jeff
    new_json=$(JSON.dump)

It's still not as trivial as javascript, but "tooth-pullingly difficult" is a little unfair.

Again: sh, not bash. If I have the freedom to specify bash, I can specify something better than bash or sh.

> but the world is more complex than what /bin/sh can see

That is blatantly false, but I get the premise behind it. Writing and maintaining complex stuff in bourne or bash is not fun or easy.

I always use bourne until certain level of complexity or awkwardness is reached. It's pretty easy to write and dead simple to troubleshoot.

Blatantly false? OK, put up. Forget `sh`'s JSON parsing, I'll make it easy on you--show me its arrays. Arrays, plural, you can't use $@ as a bail-out; I need more than one. Show me hashes. Show me sets. Show me the basic building-block primitives of software, because build pipelines are software. It's 2018. If your language can't do this stuff without babysitting, it effectively can't do it because nobody's got the time to babysit your easy-25%-of-Perl language

And yeah, I did say `sh`, because that is what you can practically be expected to have kicking around alongside `make` on a system where I can't just install something worthwhile. If I have them, then there's no reason to write much harder to troubleshoot shell scripts (and, thank you quotebarf, more likely to be incorrect) than to open Pry.

Bash, but...

> --show me its arrays

  declare -a

> Show me hashes.

  declare -A

> Show me sets.

See above

> Show me the basic building-block primitives of software,

  thing() {
     local localVar
     # Yes, it would be nice to have lexical scope as well as 
     # dynamic scope, but...
  }

Sure. All of which are clunky, opaque, and harder to work with than any other language I have used in the last decade.

And besides: if I can install bash, I can install something specifically better for whatever I'm doing instead.

Clunky is subjective -- I think we can probably agree that there are clunkier, more opaque, and harder to work with constructs in programming than a slightly different syntax for array declaration.

What are these build systems where you need to install Bash? Bash 4 was released in 2009: it's been in every major Linux distribution for at least two major versions, ditto for FreeBSD... heck, even Solaris ships with it now.

You're right, there are clunkier, more opaque, and harder to work with constructs in programming. Like `sh` arguments. And like `sh` quotebarf.

And Busybox is commonplace. Systems that include Busybox usually do not include `bash`. And so, for my purposes, if I can specify `bash`, I can also specify, say, Ruby, which--while by no means perfect--makes life much, much easier.

Speaking as an (ex- (sort of)) sysadmin,

"All of which are clunky, opaque, and harder to work with than any other language I have used in the last decade."

Wrong question. A better one: Is it more clunky, opaque, and harder to work with than every other language that's appeared in the last decade? Because no one seems to agree on what is specifically better.

I disagree that it is the wrong question. "Every other language" doesn't matter because I don't value homogeneity and I think that homogeneity of programming language is a fool's errand. I am comfortable shipping production code in most of the languages in current use; in my estimation, none of the major build systems out there are as opaque or difficult to use correctly as make/shell.

(And I am what a current sysadmin would be if we did not call ourselves "devops engineers" now.)

And you are definitely not a sysad or any sort of sysadmin. The core mission of a sysad is to build the best environment possible while restricting that environment.

You seem to be the sort of developer that developers love and the sort of sysadmin that gets fired in the first week Just my .02 after two and a half decades.

Not to say that you are entirely wrong or your approach doesn't have merit in the new world (especially SV). But it doesn't work for sysadmins and production environments anywhere but your bubble. Not yet.

Well, you're right, I'm not a sysadmin. I pervasively automate, which often sidelines sysadmins, when it doesn't make them redundant. I write code and I don't touch production machines except in extremity, neither of which apply to most (though by no means all) of the people I know who want to call themselves a sysadmin.

Anyway, the core mission of anybody touching the stack is to enable the business to achieve its goals. Nothing, and I mean nothing, more. "Restricting that environment" is appropriate in some environments, and a number of my clients bring me in to help with that. Facilitating developer velocity--and, yes, developers do tend to like me, because I'm good at this while achieving goals around security and uptime--is appropriate in, probably, more. Pays better, too, even if it shouldn't.

It's not that sysadmins cannot do the work you are rightfully proud of. If there are two basic things that differentiates your statements from those of a traditional sysadmin it is these.

1. Design. 2. Discipline.

Where these two values are dispensable long term devops and the new world shine through. I've worked in both worlds and the only mistake is assuming one size fits all.

In general you seem like an absurd sort of creature. Neither here nor there. Bragging about your facility and business velocity. Everything you claim to do sysadmins were doing in 98 and with equal velocity and adequate coverage.

There were guys like this that were there before and helped everyone along: https://www.computerhope.com/people/don_libes.htm

At the risk of being too "meta", although I agree with what I believe is your point about good sysadmins having been advancing automation (and otherwise keeping business needs in mind), I worry that you're distracting a reader from that point by what reads as an ad-hominem attack in your first sentence.

I'm still not certain what point you were trying to make with "neithere here nor there", however.

Have you SEEN how Bash array indexing works? It’s not exactly user friendly. Same with Bash hashtables (which I don’t think are really hashtables).

I love Bash and use it way more often than I should, but Bash is not a friendly language. Mocking and stubbing functions is really hard to do, which makes tests awkward, even with BATS. And you need to watch the file descriptor to which text is sent to avoid things get globes unexpectedly. And you need to properly scope your vars to avoid unexpected collisions. And everything regarding sourcing and importing dependencies. Etc.

>I use Rake (or Gulp, or whatever) because then I can use Ruby (or JavaScript, or whatever).

This is an anti-feature. When you need this, your build is too complicated.

But they're also not particularly good.

Million+ line codebases got errm made with make. In my experience people most sneering at it have far lesser demands on their tool...

Million+ line codebases were written entirely in COBOL.

The world advances.

Some were, I guess, but hardly all of them. The Linux kernel is built with Make, for example.

There are been several attempts at introducing a new build system, but they have been all shot down over introducing dependencies.

If it ain't broke don't fix it!

How many features could people have implemented or bugs fixed or general progress made instead of obsessing over tooling?

It's weird that you use the word "advances" to refer to JavaScript-based build processes.