An XMPP/Jabber echo bot written in sed | Hacker News Reader

An XMPP/Jabber echo bot written in sed

github.com

167 pointsmuch_me35 comments

"All you have is openssl, bash, dig, stdbuf and sed?"

s/sed/GNU sed/

echoz.sh also uses cut, base64, stdbuf, tr, mkfifo, tee and sort.

echoz.sh does not seem to have any "bashisms" so perhaps one does not actually need bash and other sh will suffice.

stdbuf is absent from some BSD UNIX. Below is a small script to try to unbuffer output from any program without using LD_PRELOAD. base64 is also absent from some BSD UNIX and can be replaced with

   openssl enc -a

tee can be replaced with

   sed -u w/dev/stdout

The uses of cut to isolate substrings and tr to replace characters can also be acomplished using sed.

   #!/bin/sh
 
   test $# -ge 1||exec echo usage: $0 program arguments ;
   maxarg=5;
   test $# -le $maxarg||exec echo max $maxarg arguments ;
   {  
   a=$#;
   b=$(command -v $1);
   echo -n '   #include <stdio.h>
   int main(){
   setvbuf(stdin,NULL,_IONBF,0);
   setvbuf(stdout,NULL,_IONBF,0);
   char *b['$((a+1))'];'
   n=0;while true;do
   test $n -le $maxarg||exit;
   a=$((a-1));
   test $a -ge 0||if export n;then echo;break;fi ;
   echo -n '
   b['$n']="'$1'";'
   shift 1;
   n=$((n+1));   
   done;
   echo '   b['$n']=(void *)0;
   execve("'$b'",b,(void *)0);
   }'; 
   }|exec cc -xc - -static && exec a.out

Author of xmpp-echo-bot here.

> echoz.sh also uses cut, base64, stdbuf, tr, mkfifo, tee and sort.

tee can be eliminated if needed be, the others should probably be added to the list of dependencies, thanks.

> echoz.sh does not seem to have any "bashisms" so perhaps one does not actually need bash and other sh will suffice.

Indeed, there is a PR which I have to review which fixes that.

> The uses of cut to isolate substrings and tr to replace characters can also be acomplished using sed.

I would be interested how you would do especially the tr part in sed, given that sed, even unbufferedly, operates on "lines". I use tr to replace '>' (XML delimiter) with a line ending recognized by sed to be able to operate on individual XML elements.

The uses of cut could be replaced by sed in the script indeed, although it can’t be moved into the main sed script. I made an issue for that: https://github.com/horazont/xmpp-echo-bot/issues/6

> "All you have is openssl, bash, dig, stdbuf and sed?"

I hope it is clear that this entire thing is ironic and the long list of dependencies -- especially given that bash could reasonably be used to replace sed here -- is supposed to underline that. I’ll gladly add more :).

</spoiler>

"I would be interested in how you would do especially the tr part in sed..."

There is more than one tr part. :) I tried to be careful to refer only to where the task is replacing characters.

As for joining lines, I have used the same tr -d '\12' technique quite often in the past if the input is not contained in a file. For files I would sometimes use ed scripts.

IME, sed is one of the most ubiquitous programs in UNIX-like OS. I use it daily. However when sed cannot do the job or cannot do it fast enough, I use flex to write relatively small, single-purpose programs quickly. There is a good chance they will compile cleanly on a variety of OS. Perhaps flex is as available to the user as tr, mkfifo, openssl, etc., i.e., when the user has access to all those programs she also probably has access to flex, cpp, as, ld and cc.

I would be happy to provide an example directed at the xmpp-echo-bot solution if you can provide some sample input and the desired output, ICPC-style.

In processing HTML, XML or JSON with sed I have often used tr (e.g., delete newlines, add non-printable delimiter, then replace delimiter with newline) to reformat into sed-friendly input. However, an easy alternative to using tr for this is flex.

As an example, below is a one-off/reusable HTML/XML reformatter in flex. This makes HTML/XML easier for me to read. It also makes it very easy to process with sed and other line-based utilities.

    ftp -4o 1.xml http://web.archive.org/web/20130814000845/http://zombofant.net/blog/ |a.out |less

    flex -8iCrfa 038.l 
    cc -static lex.yy.c

    cat 038.l

    #define echo ECHO
    #define jmp BEGIN
    #define nl putchar(10)
    #define ind fputs("\40\40\40",stdout)
   %s xa xb 
   xa \11|\40 
   %%
   ^\x0d\x0a jmp xb;
   \<{xa}*script nl;ind;echo;jmp xa;
   <xa>\<{xa}*\/script{xa}*\> echo;jmp xb;
   <xa>{xa}{xa}* putchar(32);
   <xa>. echo;
   <xb>\< nl;ind;echo;
   <xb>\> echo;
   <xb>{xa}{xa}* putchar(32);
   <xb>. echo;
   .|\n
   %%
   int main(){ yylex();}
   int yywrap(){ nl;}

You seem to be quite advanced in the topic of cross-unix coding.

One thing that bothers me since a few weeks is to absence of 'local' [1] from POSIX. On the one hand, I consider using 'local' for function variables good style, on the other hand that is not POSIX compliant.

I have found some Stack Overflow discussion about it [2] but that doesn't really solve the problem either. For the moment I just keep using 'local' as my scripts are unlikely to be run on anything else than a linux, but using the 'sh' shebang feels wrong while using 'local' :-/

Any ideas how to solve that dilemma?

[1]: http://www.tldp.org/LDP/abs/html/localvar.html

[2]: https://stackoverflow.com/questions/18597697/posix-compliant...

The stdbuf command exists on FreeBSD. On NetBSD, this functionality is achieved by setting the STDBUF environment variable.

STDBUF=U is one of the benefits of using NetBSD libc.

This solution is however limited to applications compiled with the libc C "standard" stdio.

If you're into these kind of tricks, here's a Bash-script IRC client in under 130 lines! https://github.com/halhen/shic/blob/master/shic The kind of things people can do with simple tools is just amazing. Makes me wonder if we're over-engineering things when we write them with hundreds of thousands of lines of code.

Bash is no more a simple tool than c or perl or lua or whatever-programming-language

Bash is "simpler" in the sense that it doesn't have proper data structures. Recent versions have something like an array, but it's still not quite an array afaik. Bash is ultimately still just a string concatenation language.

Bash has had real arrays since 2.0 (1996). Though, the `+=` operator to append to an array wasn't added until 3.1 (2005); before that you had to do `array[${#array[*]}]=word` to append.

Since 4.0 (2009) it's also had associative arrays.

It's trivial to implement an array in pure shell code without extensions. For example:

  # append ARRAYNAME VALUE
  append() {
    eval N=\"\${${1}_N:-0}\" # read [ARRAYNAME]_N
    N=$(($N + 1))
    eval ${1}_${N}=\"\${2}\" # set [ARRAYNAME]_[N]
    eval ${1}_N=\"\${N}\" # store [ARRAYNAME]_N
  }

It's rather straightforward to implement whatever data structure you want this way.

While I understand your point, looking at that code, the first word into my mind isn't "trivial".

Makes me think of this: https://i.imgur.com/1LUysfW.png

It actually is once you strip all the bash-isms. Here's what this code is really doing.

  def append(ARRAYNAME, VALUE):
    # Store the length of the array as a variable ARRAYNAME_LEN
    # Store the values of the array as variables ARRAYNAME_0, ARRAYNAME_1,...

    len = globals.get('{{ ARRAYNAME }}_LEN') || 0
    len += 1

    # Update the length.
    globals.set('{{ ARRAYNAME }}_LEN', len)

    # Add the value.
    globals.set('{{ ARRAYNAME }}_{{ len }}', VALUE)
    
    return

Nobody said that bash has pretty syntax but this kind of thing wont even phase you once you're used to all the escaping and bracket substitution rules.

For the record, the code is 100% POSIX shell. :)

Good point, I really should have said all the sh-isms.

Also, I think you're a dammed sinner for having a 1 indexed array but otherwise it's very neat :)

How can a human being evolve to this level?

https://www.gnu.org/software/sed/manual/sed.html

Holy shit there's manuals for software! ;)

Easy.

Wow, I am impressed. I didn't know you could simply use openssl and sed combined on the CLI to create a XMPP bot...

Someone should have told me that the day I was searching a simple Go XMPP client just to send a message from my server.

not to diminish this creation in any way, but if you had to create something like this might awk have been an easier tool to use, and equally available? Obviously this is just for.... i dunno someone desperately wanting to chat on a production server or the result of a geek finding an interesting challenge (maybe both) but... if you DID have to...

Somehow I don't think the point was to be practical (not that awk would have been practical either)… yes, they could have used awk. My money is on 'interesting challenge'.

Since they already use bash as a prerequisite and wrapper, they could probably re-write it entirely in bash, and skip sed altogether.

They could, but I don't think doing so would make it simpler.

Nice. This is a real Rube Goldberg machine.

Wow, 7 hours and nobody's posted https://github.com/uuner/sedtris yet.

HN, you're starting to drop the ball :P

This is impressive and amazing at the same time :p

gotta love pleonasms!

Testimonials:

This is crazy, I haven't crashed it yet! — Matthew Wild

I'm frightened. It's only two steps away from gaining consciousness. — Georg Lukas

I am simultaneously appalled and in awe. wow — Lance

With Echoz.sed, we were able to reduce our XMPP Echo server costs by 90% compared to our previous TeX-based solution. — Leon

While simple and limited, sed is sufficiently powerful for a large number of purposes. — Wikipedia

oh my god this actually works — Test

Every time I see this I crack up at: "With Echoz.sed, we were able to reduce our XMPP Echo server costs by 90% compared to our previous TeX-based solution. — Leon"

Now I really want to see the TeX based solution :)

“OK, so 10 out of 10 for style, but minus several million for good thinking, yeah?” - Zaphod Beeblebrox

(Sigh... as this is the Internet I will say here, I'm adding something, not saying it's on the page.)

"Your scientists were so preoccupied with whether or not they could, they didn't stop to think if they should."

--Dr Jeff Goldblum

Reply on news.ycombinator.com