Hacker News Reader

Comment by benhoyt | Hacker News Reader

rp looks like a very interesting and clever approach. I'll have to take a closer look. Is https://github.com/c-blake/cligen/blob/master/examples/rp.ni... the extent of rp's documentation?

View on HN

> the extent of rp's "documentation"

Unfortunately, yes. My hope was that it would be compact/small enough to be "quasi-self-documenting" to the likes of "compiler writer types". Probably not to ordinary users. (The extra asterisks are to get nice ANSI SGR escape highlighting and/or rST markup out of the auto-documentation system.) {EDIT2: Also, I would be happy to add some rp.README if you want to contribute one. You seem a great explainer. }

FWIW, I think of this kind of generation as part of the Go mentality more generally, but I am not a Go user/in that community. So, maybe that is speaking out of turn.

I also have a C version of this that I call `crp` I could provide if anyone wants (and yes, short for "crap"). C is a kind of higher ceremony language for such things { EDIT1: but even more established than awk... :-) }

Code generation is used quite a bit in Go, though I haven't seen it used quite like this before. Now you've got me thinking what an rp-like tool would look like in Go. I think the error handling and statement vs expression distinction might get in the way a bit, but perhaps you could overcome that with helper functions.

I think the design is just about 100X easier for mere mortals { in bogus "effortmeters" ;-) }, but it is absolutely true that some backend languages are more suited than others. It is also true that having real optimizing compilers (in whatever lang) as backends will always result in faster running programs (if data is big enough that amortizing optimizer costs is worth it). {EDIT: i.e. "so, trade-offs, costs, YMMV, etc., etc. }

I'll go ahead and post crp.nim here. pjmlp rightly (and often here) observes that people become very trapped in their thinking about PLs. It perhaps helps to emphasize that the whole program generator need not be in the same PL as the generated program. But for the record, I do not think C is a compiled language very well suited to these one-liner tasks. There are probably better ones than Nim, too. My point is mostly about system design. One can probably say that the best backend is probably syntactically non-noisy and easy for whoever your users are. If you are your only user, go nuts some afternoon. :-)

    import std/[strutils,os,hashes,sets],cligen/[osUt,mslice] #% exec* mdOpen split
    from cligen/parseopt3 import optionNormalize

    proc toDef(fields, delim, genF: string): string =
      result.add "char const * const rpNmFields = \"" & fields & "\";\n"
      let sep = initSep(delim)
      let row = fields.toMSlice
      var s: seq[MSlice]
      var nms: HashSet[string]
      sep.split(row, s) # No maxSplit - define every field; Could infer it from the
      for j, f in s:    #..highest referenced field with a `where` & `stmts` parse.
        let nm = optionNormalize(genF % [ $f ])   # Prevent duplicate def errors..
        if nm notin nms:                          #..and warn users about collision.
          result.add "int const " & nm & " = " & $j & ";\n"
          nms.incl nm
        else:
          stderr.write "crp: WARNING: ", nm, " collides with earlier field\n"

    proc crp(prelude="", begin="", where="1", stmts:seq[string], epilog="",
             fields="", genF="$1", comp="", run=true, args="", outp="/tmp/crpXXX",
             input="/dev/stdin", delim=" \t", uncheck=false, maxSplit=0): int =
      ## Gen+Run *prelude*,*fields*,*begin*,*where*,*stmts*,*epilog* row processor
      ## against *input*.  Defined within *where* & every *stmt* are:
      ##   *s[idx]* & *row* => C strings, *i(idx)* => int64, *f(idx)* => double.
      ##   *nf* & *nr* (like *AWK*);  NOTE: *idx* is **0-origin**.
      ## A generated program is left at *outp*.c, easily copied for "utilitizing".
      ## If you know *AWK* & C, you can learn *crp* PRONTO.  Examples (need data):
      ##   **crp 'printf("%s %s\\n", s[1], s[0]);'**         # Swap field order
      ##   **crp -w'nr % 100==0' 'printf("%s\\n", row);'**   # Prn each 100th row
      ##   **crp -b'int t=0' t+=nf -e'printf("%d\\n", t)'**  # Prn total field count
      ##   **crp -b'int t=0' -w'i(0)>0' 't+=i(0)' -e'printf("%d\\n", t)'** # Total>0
      ##   **crp 'float x=f(0)' 'printf("%g\\n", (1+x)/x)'** # cache field 0 parse
      ##   **crp -d, -fa,b,c 'printf("%s %g\\n",s[a],f(b)+i(c))'**  # named fields
      ## Add niceties (eg. `#include "mystuff.h"`) to *prelude* in ~/.config/crp.
      let fields = if fields.len == 0: fields else: toDef(fields, delim, genF)
      let check  = if fields.len == 0: "    " elif not uncheck: """
        if (nr == 0) {
          if (strcmp(row, rpNmFields) == 0) {
            nr++; continue; // {fields} {!uncheck}
          } else {
        exit(2);
      }
      while ((rpNmRead = getline(&row, &rpNmAlloc, rpNmFile)) != -1) {
        row[rpNmRead - 1] = '\0';       // chop newline
    ${6}s = rpNmSplit(s, &rpNmAlloc, row, "$3", $7, &nf); // {delim,maxSplit}
        if ($8) { // {where} auto ()s?
    """ % [prelude, fields, delim, indent(begin, 2), input, check, $maxSplit, where]
      for i, stmt in stmts:
        program.add "      " & stmt & "; // {stmt" & $i & "}\n"
      if stmts.len == 0:
        program.add "      /**/;\n"
      program.add "    }\n    nr++;\n  }\n"
      program.add indent(epilog, 2)
      program.add "; // {epilogue}\n}\n"
      let mode = if run: "-run" else: ""
      let args = if args.len > 0: args else: "-I$HOME/s -O"
      let digs = count(outp, 'X')
      let hsh  = toHex(program.hash and ((1 shl 16*digs) - 1), digs)
      let outp = if digs > 0: outp[0 ..< ^digs] & hsh else: outp
      let comp = if comp.len > 0: comp else: "tcc $1 $2 -o$3 $4" % [
                                             mode, args, outp, outp & ".c"]
      let f = mkdirOpen(outp & ".c", fmWrite)
      f.write program
      f.close
      execShellCmd(comp & (if run: " < " & input else: ""))

    when isMainModule:
      import cligen; include cligen/mergeCfgEnv
      dispatch crp,help={"prelude" : "Nim code for prelude/imports section",
                         "begin"   : "Nim code for begin/pre-loop section",
                         "where"   : "Nim code for row inclusion",
                         "stmts"   : "Nim stmts to run under `where`",
                         "epilog"  : "Nim code for epilog/end loop section",
                         "fields"  : "`delim`-sep field names (match row0)",
                         "genF"    : "make field names from this fmt; eg c$1",
                         "comp"    : "\"\" => tcc {if run: \"-run\"} {args}",
                         "run"     : "Run at once using tcc -run .. < input",
                         "args"    : "\"\" => -I$HOME/s -O",
                         "outp"    : "output executable; .c NOT REMOVED",
                         "input"   : "path to read as input",
                         "delim"   : "inp delim chars for strtok",
                         "uncheck" : "do not check&skip header row vs fields",
                         "maxSplit": "max split; 0 => unbounded"}, cmdName="crp"

Oh, btw, all that "Nim code for XYZ" in the crp,help={} section at the end should be "C code". Oops. { Sorry. I mostly just ported the rp.nim to crp.nim to time how fast tcc could make a pure C backend run and didn't change that part. }

I should have said, those help strings at the very bottom of the file are also part of the "documentation" to the extent the variable names in the proc signature are not self explanatory.

Reply on news.ycombinator.com