How `curl -L cueup.com/jobs` works | Hacker News Reader

How `curl -L cueup.com/jobs` works

tech.cueup.com

84 pointsdanicgross35 comments

I dumped it into a file out of habit, which sort of ruined the effect. I did wonder why a ~300 line textfile took so long to DL, though.

Any hacks you can think of to detect this from the server?

Start out by sending dummy data as fast as you can. Once you've sent a few TB you can be relatively confident that it's going to a terminal instead of a file (which would have filled up the disk by now). Then you can start slowly sending the real data.

Curl is not going to send different request headers to the server based on whether its stdout is redirected to a file.

I'm afraid my system does not have yet support to http protocol

This server does not apparently has any service listening on port 23 so unfortunately I can't telnet to it (I'm not sure what's this /jobs as well, is this a Gopher page? oh well...)

I hoped this would work:

    $ telnet cueup.com 80
    GET /jobs HTTP/1.1
    User-Agent: curl/7.24.0 (x86_64-apple-darwin12.0) libcurl/7.24.0 OpenSSL/0.9.8r zlib/1.2.5
    Host: cueup.com
    Accept: */*
    <blank line>

but it redirects to Location: https://www.cueup.com/jobs, so this is necessary:

    $ openssl s_client -connect www.cueup.com:443
    GET /jobs HTTP/1.1
    User-Agent: curl/7.24.0 (x86_64-apple-darwin12.0) libcurl/7.24.0 OpenSSL/0.9.8r zlib/1.2.5
    Host: www.cueup.com
    Accept: */*
    <blank line>

And there you go (^D to stop).

Unfortunately things get munged up since it's using Transfer-Encoding: chunked, so there's the chunk size and blank line in between each content line.

Bonus points for cueup.com if they update the thing to not chunk it if User-Agent: telnet (I did test this and it doesn't work)

Port 22 is SSH. Telneting to it wouldn't do much good aside telling you which sshd was listening.

I'm also a little confused by what you mean when you said your system doesn't support "HTTP protocol", curl has been a staple for Unix and Linux for over a decade. Or are you trying to run curl from cmd.exe in windows?

I meant port 23, sorry

And maybe you're too young to understand the joke ;)

Possibly. I know good jokes are ruined if they have to be explained, but would you mind on this occasion :)

Well, here goes:

Once upon a time (around 1995, first time I tried this 'internet' thing) the internet already existed. But this newfangled http protocol was not very popular (it existed since 1991)

You see, Curl was created in 1997, Wget in 1996

The way to access systems was through telnet mostly. Some modern clients, like lynx, also supported Gopher http://en.wikipedia.org/wiki/Gopher_(protocol)

(Yes, lynx got http support at a later time)

So, their choice for a page readable from a terminal ends up being kind of an Anachronism =)

Oh, I know all that (I was building websites in the mid 90s, so I'm old enough to remember a life before the WWW).

I assumed you were referencing an old meme or a famous quote from some notable UNIX greybeard. I hadn't realised you were just joking about being old (that wasn't clear from your original comment).

Sorry mate, I just misunderstood the context of your joke. (my epic lack of sleep probably didn't help)

Ah it happens, don't worry =)

Saving the explanation, try this:

    $ telnet towel.blinkenlights.nl

I don't think there's a single nerd on the planet who hasn't already seen that.

OP here. I'm curious to learn of any other creative uses of ANSI and curl. Does anyone know of websites that use it as an easter egg?

Not curl, but I did something similar for a particularly dramatic death scene in a MUD 15 years or so ago. The hardest part was shutting up all of the normal MUD chatter that the players get (OOC chat and the like).

One thing that I often use is

  curl ifconfig.me

But this is so damn cool!

since you're only dependent on dumping stuff to the terminal, I think Telnet is comparable. Did you see the Blinkenlights Star Wars?

There are some pretty cool Telnet BBSes still running:

http://www.telnetbbsguide.com/brieflist.asp

It's a good way to be suddenly overpowered by nostalgia. I feel this strange desire to PEEK and POKE, and perhaps to call interrupt 13h.

Slashdot used to stick random Futurama quotes in their http header. I'm not seeing it anymore. curl -sI http://slashdot.org

I just added one to http://poyo.co .. thanks for the idea!

Maybe it's just me (using PuTTY), but it is next to impossible to read while it scrolls up my screen. Once it's done scrolling I can't see most of the listings because by default my terminal doesn't save that many lines of history.

Awesome idea though

I'm pretty sure there's a setting in PuTTY to increase the size of the scrollback buffer. Memory is cheap these days.

With HTTPie (master):

   http --pretty=none https://www.cueup.com/jobs User-Agent:curl

(`--pretty=none` is needed because the server incorrectly sends `Content-Type: text/html`)

Works with Wget too:

wget -q --user-agent="curl" -O - cueup.com/jobs

Reminds me of an old dialup BBS. Can I post a comment on the wall?

Where are these colorMarkdown and htmlToMarkdown functions?

Sadly there seems to be no MIME type for ANSI art. Of course this should depend on the "Accept" header and not the User Agent. But it seems infeasible.

Is anyone else unable to see this due to tech.cueup.com not resolving?

You know the worst kind of DNS problem? The kind that only affects some people. Yeah, I can't resolve the domain name either. Ugh. For now, the Coral Cache hack works:

http://tech.cueup.com.nyud.net/blog/2013/03/06/how-our-termi...

EDIT: Looks fixed now.

It might help to add Google's DNS to the list of DNS servers, e.g. on my Ubuntu, I sometimes add "nameserver 8.8.8.8" to the end of my "/etc/resolv.conf" file.

Your /jobs link in your sites footer is 404ing

This is a cool idea, but anyone who needs "how it works" explained is really not the target audience.

Honestly, fuck anyone who does this for a job posting.

Why? There's a normal (nice looking) page for other user agents. You haven't lost anything.

Reply on news.ycombinator.com