flex -8iCrfa <<eof
int fileno (FILE *);
xa "\15"|"\12"
xb "\15\12"
%option noyywrap nounput noinput
%%
^[A-Fa-f0-9]+{xa}
{xa}+[A-Fa-f0-9]+{xa}
{xb}[A-Fa-f0-9]+{xb}
%%
int main(){ yylex();exit(0);}
eof
cc -std=c89 -Wall -pipe lex.yy.c -static -o yy045
Yahoo! serves chunked pages
printf 'GET / HTTP/1.1\r\nHost: us.yahoo.com\r\nConnection: close\r\n\r\n'|openssl s_client -connect us.yahoo.com:443 -ign_eof|./yy045This is what it produces for me when I run `lexit.sh us.yahoo.com` - https://stuff-storage.sfo3.digitaloceanspaces.com/ee.txt
The compiled program is only useful for filtering chunked transfer encoding on stdin. Most "HTTP clients" like wget or curl already take care of processing chunked transfer encoding. It is when working with something like netcat that chunked tranfser encoding becomes "DIY". This is a simple program that attempts to solve that problem. It could be written by hand without using flex.
The "gibberish" is GZIP compressed data. "yy054" is a simple filter I wrote to extract a GZIP file from stdin, i.e., discard leading and trailing garabage. As far as I can tell, the compressed file "ee.txt" is not chunked transfer encoded. If it was chunked we would first extract the GZIP, then decompress and finally process the chunks (e.g., filter out the chunk sizes with the filter submitted in the OP).
In this case all we need to do is extract the GZIP file "ee.txt" from stdin, then decompress it:
printf "GET /ee.txt\r\nHost: stuff-storage.sfo3.digitaloceanspaces.com\r\nConnection: close\r\n\r\n"|openssl s_client -connect 138.68.34.161:443 -quiet|yy054|gzip -dc > 1.htm
firefox ./1.htm