Convert curl commands to Go code | Hacker News Reader

Convert curl commands to Go code

mholt.github.io

193 pointspythonist43 comments

Hey HN, I wrote this to scratch a perpetual itch after reading enough API documentations and re-writing basically the same code over and over. Right now it only supports the most common flags that I saw. As far as I'm concerned the project is finished, but PRs are welcome to cover any glaring omissions or fix bugs.

Someone also showed me this tool which does the same thing but supports more languages and options: https://shibukawa.github.io/curl_as_dsl/

Very cool. Flags I commonly see/implement (curl to Python or Ruby usually) include -o (download) and -k (ignore ssl warnings -- you might want an warning comment if you implement that one).

Great work, mholt!

He also wrote json-to-go[0]--copy/paste JSON and it generates a matching Go struct. This ends up being extremely useful and saves a lot of time for extensive JSON responses.

[0]: https://mholt.github.io/json-to-go/

This looks actually awesome, I might end up using this to then throw some regexp’s at the result to convert it into usable Java.

I already have a small python script that compiles C++ headers into Java interfaces (for services with RPC sync).

I use this all the time! Thanks for your work op

Thank you :) Glad to know it helps more than just me.

This is awesome. Combined with Chrome Developer Tools' "Copy request as cURL" feature, you can reverse engineer some service and turn it into Go code so easily.

Agreed! Burp Suite also has the ability to copy a request as a cURL command, which can definitely help with reversing mobile device API's as well.

So how about a variation that takes a curl command and writes Go http server code that would accept that request?

Speaking generally, it's more difficult to do much useful with that, because it's hard to do much more than route on the URL. You could prototype code that merely looks at the mentioned headers, but it's getting kinda silly at that point.

More specific to JSON, you may be looking for something like https://mholt.github.io/json-to-go/ (by the same author) or https://github.com/ChimeraCoder/gojson (which I've used myself).

(I find that what I pay in using a static language like Go to manipulate JSON and having to use a tool like that often comes back to me pretty quickly when I take the tool output as a template and start turning all the structs into Real Objects (TM). The dynamic languages are pretty good at taking JSON and yielding a simple pile of dict/map/array/strings/numbers/etc., but if you want to get real objects back out of them the advantage over Go evaporates. Not because the dynamic languages make it "hard", but just because with one of these tools, both Go and the dynamic solutions are roughly as easy.)

Couldn't you write a program that does that trivially by writing Go http server code that accepts any request?

Ah. That's what makes it useful. Writing

   curl -X POST -H "Content-Type: application/json" -H "Authorization: Bearer b7d03a6947b217efb6f3ec3bd3504582" -d '{"type":"A","name":"www","data":"162.10.66.0","priority":null,"port":null,"weight":null}' "https://api.digitalocean.com/v2/domains/example.com/records"

by hand would seem to indicate you're using the wrong tool for the job. But if other tools use curl commands as a notation for HTTP requests, curl command to programming language conversion has a real use case.

The "copy as curl" feature is very useful when doing penetration testing of a web app, because you can quickly iterate on what requests you're sending. It also helps to download large files if, like me, you don't trust browsers to download your files properly (or if you want to download something from a server).

APIs aren't the only thing HTTP is used for. Although, you could stick curl commands in shell scripts as well to have dumb API scripts.

There are use cases for every feature.

Given the connection with Chrome Developer Tools' "Copy request as cURL" I can see the usefulness of this in a general sense; however, I was personally somewhat sad to see that it doesn't actually convert to code that makes use of libcurl.

As such, the resulting project is very limited in exactly which cURL commandlines it is capable of converting, and, particularly, it is only able to handle HTTP connections (unless I missed something).

A generator that could generate libcurl compatible code would have the full power of cURL!

This is really neat. There's, I don't know if it's a niggle or a bug or an interface issue, but well, let's call it a usability bug.

If your copy to curl request in chrome accepts gzip and/or deflate, then your tool will add an "Accept-Encoding" Header. So far so good.

In go 1.5.1 at least, from my cursory testing, the runtime will not auto-decompress a bytestream if you've manually specified an accepted encoding.

On the other hand, if you leave it off, it will offer gzip to a server and auto uncompress on receipt.

I'm still not sure this is a bug, but I definitely commented out my accept encoding line in my sample code because I don't want to manually decompress every response.

At any rate, thank you for making this; I'm using it with a slack integration right now.

Another alternative is Postman [0]. Supports C, Java, Go, Ruby, Swift, etc. Available as a Chrome app. Not really the same (doesn't convert from one to another AFAIK), but it gives you a common interface.

[0]: https://www.getpostman.com

Thanks for the mention. Postman can import curl commands too and then of course convert to all these languages. We are opening this up so that converters can be added/coded dynamically.

Note that by default, curl itself can also output C code version of the CLI command as well (yay)

It's not that easy to parse this curl command. He needed 400 lines (https://github.com/mholt/curl-to-go/blob/gh-pages/resources/...). Most libraries out there do this job, but are focused on parsing from environment variables instead of a string argument, because the programs that needs this usually are CLI programs. That's what I found when doing the parse in ruby which basically has only OptParse. There is an opportunity here. Projects like these that receives a command input are very useful.

It turns out options parsing is fundamentally hard. It looks simple at first, but then it turns out that if you want to support everything anybody could possible want, you end up with some too complicated to use. Nor is it obvious which "middle ground" is correct. It's a perfect environment to produce exactly what we get, which is a bajillion options parsers for every halfway mature language in the world.

Also, that's not 400 lines of parsing code, that intermixes "parsing" and generation all together.

Mind you Go is not a good language for writing parsers in. (It isn't especially bad, like, say, C, but it lacks almost every conceivable feature that could make parsing code anything other than very, very verbose.)

Just to expand on this: options can be position-dependent, and treat options as an algebra (a --and b --or c); or, just executed in order to eg change directory (-d dir1 a b c -d dir2 x y z).

Basically, by treating the arguments as a stream of tokens, it can be parsed with the same power used for parsing the source of programming language.

Yet for many uses there are natural complexity-levels for option parser tools (like position-independent boolean options, and options taking one argument), hence the many tools for many niches.

And they are easy to write, so everyone writes one.

\tangent Wouldn't it make sense for this to be a library? Though most of the py looks so simple, it already has the libraries.

Or, to parse the curl command string at run time, so it can easily be human-read, as a curl command?

But I guess the idea is to generate a start-point template, which can then be customised for whatever is needed.

It would be nice for a tool to combine both - so new code is attached to the curl command string... but not very readable... and not very flexible (I bet the next thing you wanted to add would be something that hadn't been considered...)

A couple of these tools have sprung up:

JSON: https://mholt.github.io/json-to-go/

CURL: https://mholt.github.io/curl-to-go/

SQL: https://github.com/knq/xo

The JSON tool especially has saved me a lot of time.

great idea - i wish this existed for every language !

Would love the same thing for python!

http://curl.trillworks.com/

Awesome, thanks!

I'm working with the PayPal API right now (send your condolences) and all their documentation is in the form of CURL requests. Just finished converting it to python requests commands and now this gets posted.

TIL'd.

Thanks, if this wasn't posted I would've written that version myself.

I built this! It looks like there's a lot of excitement about generating Go code. If someone wants to send a pull request incorporating the Go generator, I'll happily accept it.

Looks like backtics aren't sanitized

    curl -XPOST httpbin.org/post  -d '{"sneaky backtick": "`"}'

produces

    body := strings.NewReader(`{"sneaky backtick": "`"}`)

Super minor point: according to the curl manpage, -d data implies POST, so you don't need -X POST in the examples

Go question. Is:

    if err != nil {
        // handle err
    }

Required, or can you also do:

    if err {
        // handle err
    }

There's no implicit comparison (if that's the CompSci name) in Go. You have to explicitly compare (!= is required)

Nice! Small nitpick: "-A" doesn't seem to be supported.

Yeah I don't see it in the list at the top of the page. But the author says that he'd be happy see a pull request for common options that aren't supported yet.

Go is really verbose but this is really great.

It's interesting that all the error handling branches are left empty. What does production Golang code do in that case as a common idiom?

Varies widely. Usually you just return the error to let it bubble up to the first caller with enough context to handle it properly. You can also wrap the error message in a string that helps trace the path of the error. If it gets up to main(), you log it and maybe terminate (some errors are fatal, others aren't).

One of the (possibly contentious) principles in Golang is that each error needs to be addressed individually with an appropriate response to the particular case in question.

In practice most Golang errors are handled either:

1) Ignored

2) Logged and swallowed

3) Returned at the point it happened.

4) Fatally exits the program.

One of the "features" of the errors as return codes is that you don't get surprising results that come from the goto style jumps that happen with something like exceptions (unrolling state, etc).

That's not really true. It's certainly an idiom in Golang to handle errors individually, but it's not a principle.

The principal is that errors are values like everything else, and you can write code to handle them however you'd like. See, for instance, the "write" example in this post:

https://blog.golang.org/errors-are-values

There are libraries that people use (sqlx might be an example) that are more like that example than the if-error-return stuff here.

Just to expand on this for other's benefit, it's not idiomatic for errors to result in panics; in general Go code will soldier on. If you're expecting code with trouble to 'throw' an exception and puke, you're in for a bad time, or at least a confusing one.

>The language's design and conventions encourage you to explicitly check for errors where they occur (as distinct from the convention in other languages of throwing exceptions and sometimes catching them)

http://blog.golang.org/error-handling-and-go

Reply on news.ycombinator.com