HTTP/1.1 pipelining example: DNS-Over-HTTPS

As a demonstration using only standard utilities, the three scripts below 1.sh, 2.sh and 3.sh will

1. accept a list of domains and output the required Base64URL-encoded DNS request

2. make a single HTTP connection to retrieve all the responses in binary format and write them to a file

3. convert the binary file to text, suitable for manipulation with text-formatting utilities so the DNS data can be added to HOSTS file or a zone file

bindparser is from curveprotect project; it converts BIND-style output from drill to tinydns zone file format.

Most HTTP servers on the internet do have pipelining enabled, sometimes with a max-limit of 100. The three examples listed below were pipelining enabled with max-limit greater than 100 last time I checked.

Example usage:

   1.sh < list-of-domains > 1.txt
   2.sh 001 < 1.txt > 1.bin 
   3.sh < 1.bin > 1.txt
   bindparser 1.txt > 1.zone
   x=tinydns/root/data;cat 1.zone >> $x/data;cd $x;
   awk '!x[$0]++' data > data.tmp;mv data.tmp data;tinydns-data;cd -

   # 1.sh
   #!/bin/sh
   while IFS= read -r x;do printf $x' ';
   drill -q /dev/stdout $x @0.0.0.0 a|sed 's/;.*//'|xxd -p -r \
   |openssl base64|sed 's/+/-/g;s|/|_|g;s/=.*$//'|tr -d '\n';echo;
   done

   # 2.sh
   #!/bin/sh
   case $1 in
   "")sed '/;;[0-9]*/!d' $0;
      echo usage: $0 provider-number \< output-from-script1.txt;exit
   ;;001)x=doh.powerdns.org;y=1              
   ;;002)x=ibuki.cgnat.net;y=1               
   ;;003)x=dns.aa.net.uk;y=1                 
   esac;
   case $y in
   1) sed 's/\(.* \)\(.*\)/GET \/dns-query?dns=\2 HTTP\/1.1\r\nHost: \1\r\n/;
       $!s/$/Connection: keep-alive\r\n/;$s/$/Connection: close\r\n\r\n/;' \
       |socat -,ignoreeof ssl:$x:443,verify=0 > 1.bin
   esac;
   3.sh < 1.bin

   # 3.sh
   #!/bin/sh
   while IFS= read -r x;do sed -n /${1-\.}/p\;/${1-\.}/q|xxd -p|drill -i /dev/stdin 2>/dev/null;done