HTTP Toolkit
httptoolkit.tech
httptoolkit.tech
A modern REST/HTTP UI client is expected to provide one or more of the above.
I don't know what licence Postman uses. A quick search on GitHub didn't turn up the source of the app on the Postman Labs page. Insomnia is MIT [0], so it could still be forked if Kong got decided to stop supporting the free version.
Either way thanks for the recommendation/reminder for Insomnia, it looks so much better than the current version of postman.
"Insomnia is available for Mac, Windows, and Linux and can be downloaded from http://insomnia.rest/download/"
https://github.com/Kong/insomnia/commit/23cca8c42b80aa6d9de2...
And were recently acquired, so likely to continue down that path: https://blog.paw.cloud/paw-joins-forces-with-rapidapi/
My previous company was an early & very heavy user of Paw - with thousands of endpoints in our project. Unfortunately at that scale, it has some major problems, particularly with syncing.
A friend of mine has been working on https://getbeak.app to try and address those problems, but it's quite early stage still
I like it because you can just write a text file with the request and any comments you need around them, and... being just text, it's so easy to manipulate.
When I need scripts for special auth I fall back to postman though, haven't digged enough to see if I can make it work with that addon (or any other one).
Also there's Thunder Client [1] which I haven't tried but apparenly has more features.
EDIT: references
[0] https://github.com/Huachao/vscode-restclient [1] https://www.thunderclient.io/
I suspect that Vim, Sublime, or Notepad++ should likely already have a good equivalent of it, too.
If you’re a lone wolf developer who prefers a GUI - fine. But then, why pay up for the «pro» service?
So basically, as they expect to be able to write their own requests and save / share them, they would have to either directly interact with git (which they'd rather avoid - at all) either have the "real devs" provide some kind of functionality similar to postman but backed by git. Needless to say, we're not in the business of developing a postman competitor, so devs have other things to do.
I wanted something where I could store and share my requests over git instead of creating some random account. I also wanted the api description to be text not a UI+JSON blob.
I also made this https://github.com/dylanowen/sublime-dot-http
The best alternative to Postman is curl coupled with jq and xmllint.
Well known for the CLI, they now have a web/desktop client as well.
It is a desktop app though, so you can't download it usefully on mobile. If you visit the landing page on mobile, it just offers to take your email address and send download links to your computer to get you started (and as an easy "bookmark this for later" option).
That sends one follow up "Did you try it? Let me know what you think" email a week later, then it deletes your email, that's it. It's never shared elsewhere, it doesn't sign you up to the mailing list, there are no "great new offers", nada.
Meanwhile, if you're on desktop it just shows a download button directly. After that starts it asks if you want to join the mailing list for updates, but you don't need to - the download starts immediately.
Which is probably just to say after being burned too many times by bad actors, folks will start to group good actors in the same lot for similar patterns, even if the intent and design is better.
I haven't seen the source for the .deb package, but in theory it could add a system cert at installation time. I don't know if it does, though.
It actually doesn't install system certificates at all though. It doesn't change any system configuration whatsoever, and it doesn't need any admin/root privileges. The deb package doesn't do anything different to any others.
That's because the key differentiator of HTTP Toolkit vs Fiddler/Charles/mitmproxy etc, is that it provides targeted interception, rather than intercepting your entire system at once.
That works by injecting cert & proxy config into a single browser window, intercepting specific Android apps, targeting individual Docker containers etc. That way you get much less noisy intercepted traffic for your debugging, and you can freely add rules to rewrite/break traffic without interfering with anything else.
You can even open two HTTP Toolkit windows on one machine, and intercept things separately into each one.
If you want, you can still do the normal steps to do full system interception manually if you'd prefer that, but by default it uses entirely transient and permissionless targeted interception instead, and that's almost always the better approach.
In short, most of the time you need to either:
- Connect an Android emulator or a rooted device to ADB, in which case HTTP Toolkit can do totally automated setup for you.
- Use a non-rooted device, and make some minor config changes to the target application (trivial if it's your own application, slightly more difficult if it's not).
That handles 99% of Android apps, which usually don't actually pin certificates - they generally rely on Android's built-in non-modifiable system certificate store instead.
Lots more detail on how this all works here: https://httptoolkit.tech/blog/intercepting-android-https/
For apps that really do manually pin certificates, I've also written a general purpose Frida script that covers most cases out of the box. There's a full guide with more detail here: https://httptoolkit.tech/blog/frida-certificate-pinning/. And if even that doesn't work, I've also written a "reverse engineering an Android app from scratch so you can write you own Frida script" guide here: https://httptoolkit.tech/blog/android-reverse-engineering/
Here are steps: Download frida script from httptoolkit server and binary from frida github repo and download httptoolkit app in andriod. Here are my notes.
``` # Copy the server to the device adb push ./frida-server-$version-android-$arch /data/local/tmp/frida-server # ^Change this to match the name of the binary you just extracted
# Enable root access to the device adb root
# Make the server binary executable adb shell "chmod 755 /data/local/tmp/frida-server"
# Start the server on your device adb shell "/data/local/tmp/frida-server &"
pip3 install frida-tools frida-ps -U frida --no-pause -U -l ./frida.js -f com.appname
# derived from https://httptoolkit.tech/blog/frida-certificate-pinning/ ```
You can download server binaries from here https://github.com/frida/frida/releases
How hard would it be to implement a network rule "if http://abc.com/def is requested reply with this data: ..."?
Or is it possible to inject something like this on the fly into Apache?
Would be very nice to mock end-2-end tests.
I imagine you expect something like a CGI script with mod-rewrite, but your comment only actually requires plain Apache. A network rule of "if URL is requested reply with this data" is implemented by putting a file at the expected place.
> only actually requires plain Apache
As a result, to implement HTTPS interception / rewrite / injection you need some degree of modification of the application itself. The "minimal" way is to add a new TLS certificate to the certificate trust store the application uses that is marked as "allowed for every domain" (that's what Burp suite does). It seems that HTTP toolkit does it differently for the browsers it supports, probably a plugin/extension added to the browser that alters the traffic after the TLS block (HTTPS is HTTP over TLS)
It's MIT-licensed, and you can build an automated HTTP/HTTPS rewriting proxy using that in a handful of lines of JS, and script any kind of transformations or inject any responses you like.
There's a general guide to getting started here: https://httptoolkit.tech/blog/javascript-mitm-proxy-mockttp/.
For the more general interactive testing/debugging case, you can also use HTTP Toolkit itself (it has a rules builder for this kind of thing) but if you're building automation you should just use the internals directly, they have exactly the same capabilities. HTTP Toolkit just provides a UI and convenient interception setup tools over the top.
You might want to consider migrating from node-abort-controller to native AbortController by the way.
Would you consider using the more correct terminology "Fake" instead of "Mock"?
In software testing, the two terms mean different things. A "fake" is a thing that takes the place of a service; you call it and it returns a fake result. A "mock" is a thing that enforces expectations of its caller; you specify a sequence of calls that you expect, then you run the caller and it verifies that the calls occurred in the manner and order expected. They are used for different purposes, and it's useful to have the clarity of two different terms for two different things.
In particular, it is usually a better testing practice to use fakes instead of mocks. But now there are some testing libraries that create mocks (and call them mocks) and other testing libraries that create fakes (and also call them mocks), which has confused the issue and made it harder to speak clearly about a fairly important testing strategy decision.
What HTTP Toolkit does is produce fake replies to HTTP requests. So "fake" is the best term for what it does, and thus I make this earnest plea to use that term to distinguish it from mocking, which it doesn't do.
I'd be interested both in why I'd prefer the open source httptoolkit and pro?
There's certainly examples that does not use openssl/gnutls (and compatible friends) - but it's a bit of a stretch to say most stuff doesn't support it?
Most (all) Linux distros also have an easy way to add a system level cert, without messing with system managed certs. And AFAIK it's straightforward to install custom certs in the windows cert store as well.
> MITM proxy doesnt include any builtin way to install a system certificate.
Absolutely fair point of comparison. Most tls stacks will allow you to do this - via environment vars - so you can set a cert path for openssl when launching a ruby (or nodejs?) process, and things will just work.
But you then need to know mitmproxy and your tls stack.
Compared to mitmproxy, HTTP Toolkit:
- Has fully automated setup for most browsers, docker containers, Android, all Node.js/Ruby/Python/PHP/Go applications run from intercepted terminal windows, all JVM processes, any Electron apps etc etc. Some of these automated setup steps are very difficult to do manually (e.g. intercepting Android devices, where you can't normally install your own certificates nowadays, or intercepting Node.js, which completely ignores system proxy settings) so this can make a huge difference in non-trivial case.
- Supports targeted interception (intercept just one app/container/browser window) whilst all mitmproxy's manual setup steps are generally focused on helping you intercept your whole machine at once. Intercepting the whole machine means very noisy interception and means that rewriting traffic interferes with all other usage of your machine. Targeted interception means you can do neat things like run two HTTP Toolkit instances independently at the same time, and means you don't need root privileges or permanent configuration settings.
- Has a VPN app for Android, which allows it to capture traffic even if it tries to ignore proxy configuration, means you don't have to manually edit and delete Android proxy settings, and which can automatically tunnel traffic over ADB connections, so you can intercept a device connected via ADB even if its not connectable over the wifi from your computer.
- Has generally friendlier UI & UX (imo). For example, mitmproxy uses a unique custom syntax (https://docs.mitmproxy.org/stable/concepts-filters/) of special characters to define matching & rewriting rules, or requires you to write a full python script. HTTP Toolkit lets you click 'new rule' -> 'GET requests' -> 'match regex <blah>' -> 'then reply with <blah>', and then immediately start injecting automated fake responses. From HTTP Toolkit you can then build named groups or these rules, and import & export them (as JSON) to build libraries you can share with your colleagues.
- Provides lots more background information automatically: e.g. built-in documentation for all standard HTTP headers, body autoformatting for lots more formats, syntax highlighting, code folding, regex searching etc of request & response bodies, plus 'this is how and why this response could be cached' caching explanations, OpenAPI-powered docs for recognized endpoints on 1400+ APIs, etc.
- Includes advanced features to do things like exporting requests as ready-to-use code for various languages & tools, or automatically testing the performance of different compression algorithms on a given response body.
- Is more easily scriptable for automation & end-to-end testing, because all the HTTP-handling internals are usable as a standalone open-source JS library: https://github.com/httptoolkit/mockttp
That said, mitmproxy has been around longer, it's definitely more mature, and it was a big inspiration in many places. It's a great project! It does have some advantages of its own:
- If you strongly prefer a CLI interface, mitmproxy is very focused on that, and HTTP Toolkit is not. HTTP Toolkit could support that too in theory (the backend & frontend are independent) but it definitely doesn't right now, and it's not high on my todo list (contributions welcome though!)
- Mitmproxy is primarily scriptable in Python. You can build automation around HTTP Toolkit's internals using mockttp, but that's JS, and it's mostly usable standalone right now, rather than integrated into normal workflows within the app. If you want very complex scripted rules, mitmproxy has a few more options right now, and lets you do things in python instead of JS, which some people will prefer.
- WebSocket debugging - this is coming for HTTP Toolkit soon, but it's not available today. WebSockets get passed through fine, but they don't appear in the UI, and you can't set up mock rules for them.
> I'd be interested both in why I'd prefer the open source httptoolkit and pro?
There's a list of Pro features at https://httptoolkit.tech/pricing/. Note that it's all open source, even the Pro code, everything.
The general idea is that everything you need to intercept, inspect and manually fiddle with traffic is totally free. Anything optional that most users don't need, but which is helpful for advanced usage or enterprise use cases, requires Pro.
I am very delighted to use software like httptoolkit.
The only issue with httptoolkit is electron but it isn't problem for me because I can always run it in browser <3.
I understand that intercepting HTTPS might be a bit complicated for iOS and still think this is a great project though!
Otherwise, yeah it's possible, but again, it has always been possible with more complicated (compared to this) tools anyway.
In the meantime it's still totally possible to intercept iOS devices, but you just have to do the initial setup manually unfortunately.
b) Export request is a very essential feature that's available for free in either of these alternatives and its pay-walled here.
c) Intercept request in a terminal very badly broken. It broke (go get) and probably breaks others. I expect it adds an environment variable (which can be ignored by an application) or uses LD_PRELOAD (which doesn't work in statically linked applications).
Other than that, it functions like you would expect it to. Worked out of the box for Firefox and curl.
For the terminal, there's a few mechanisms, but environment variables are the catch-all there, yes (full list: https://github.com/httptoolkit/httptoolkit-server/blob/maste...). Those do work for most cases, but it is absolutely not a hard guarantee for applications that actively ignore standard proxy configuration (handling that is very hard, and definitely out of scope here).
Go does generally observes `http_proxy` correctly by default in other cases I've tested, so this vert simple code from the test suite is automatically intercepted for example: https://github.com/httptoolkit/httptoolkit-server/blob/maste.... Very happy to look into any failing cases you can share.
Here's what I did.
* Intercept Tab > Fresh terminal.
* In the terminal, do your usual stuff, it intercepts curl, etc.
* If you try to use go's package manager, example: `go get golang.org/x/oauth2` It errors out with
``` go get: module golang.org/x/oauth2: reading http://127.0.0.1:8000/golang.org/x/oauth2/@v/list: 500 Server error ```
Ideally it shouldn't break the application, just ignore if it can't intercept.
> absolutely not a hard guarantee for applications that actively ignore standard proxy configuration (handling that is very hard, and definitely out of scope here).
I encountered a usecase where this was needed and LD_PRELOAD trick (used by proxychains) etc failed because the application was statically compiled. I ended up using https://github.com/hmgle/graftcp which somehow manages to force tcp traffic through a socks5 proxy.
> If you try to use go's package manager, example: `go get golang.org/x/oauth2`
I just tested, and `go get golang.org/x/oauth2` seems to work fine for me, I can see all the requests being happily intercepted immediately: https://imgur.com/a/Cb1y9Q2
Can you see the 500 in HTTP Toolkit, and any more info there (in the body or as an error at the top) related to that? Or can you see a "certificate rejected" message? If nothing turns up there at all then yes, something must be overriding the proxy configuration.
Maybe you have some other Go package manager configuration that conflicts with this? I'd be very interested to know about that if so, I'm sure there's others with the same thing. It's always very hard to know if my configuration is representative of normal devs for any given language/tool.
Probably best to debug this outside of a HN thread though :-). You can file a proper issue about this at https://github.com/httptoolkit/httptoolkit/issues/new, I'd love to know what's going on there and get this fixed.
> I ended up using https://github.com/hmgle/graftcp which somehow manages to force tcp traffic through a socks5 proxy.
Really interesting, thanks! I'll look into that.
For doing proxies, there is https://proxyman.io, which I think is also native (haven't used it a lot, not sure)
... and as a result works only on Mac, so not usable for close to 90% of people. Every coin has 2 sides.
* https://blog.paw.cloud/paw-joins-forces-with-rapidapi/
* https://twitter.com/luckymarmot/status/1359541539161190401
Also, Setting up the desktop app vs the browser extension, the difference is way huge. It's way easier to work with browser extension when it comes to your intercept & modify browser traffic.
As of Now, We have started to get requests to scale Requestly to other platforms to debug Mobile & Desktop app traffic too hence we have recently started to work on the desktop app (https://requestly.io/desktop) (It's still in beta but usable). Here's a video a friend created sometime back - https://www.youtube.com/watch?v=LUqlA9j9Lx4&t=1s which explains how you can modify headers using Requestly desktop app.
We'll try our best to solve the problems with Requestly that still exists in Charles, Fiddler and similar tools.
Now saying it should be just cheaper in general, or perhaps more tiers, sure.
1. Due to demands from marketing/sales, the supplier tends to increase versions for what is actually a minor feature, just to justify a new payment.
2. A subscription process is the most honest way of selling software. Jetbrains is a good example of this, where you get to keep your "fallback" version when you stop the subscription.
3. You often have to wait a long time for the next release, if you don't want to register in an "early access" program with versions that break constantly (basically providing free test resources).
4. You need to justify the new version to your boss, so that you can get it covered and start the approval process higher up in the hierarchy.
5. Approval processes in large Enterprises are often complex and time consuming. This not only applies to approving the payment, but often need to involve central IT. With a subscription model, this is only done once.
2. JetBrains charges me $12 a month ($149 annually) for its entire suite of software. You think this tool by itself is worth $120 a year? For a personal license?
3. Developer's problem, not mine.
4. This is not an issue everywhere. You work with penny-pinching mongoloids.
5. This is not an issue everywhere. You work with penny-pinching mongoloids.
Also you
>You think this tool by itself is worth $120 a year? For a personal license?
Have you considered this is not an issue everywhere?
trying to trick people into giving you money forever is a paradigm shift that really needs to be rethunk.
Asking me, the end-user, to commit to $14 a month in perpetuity for this type of software is a big stretch in my opinion. I understand it for the Team tier, but for the personal tier, it doesn't make sense. $5 a month? Maybe.
Maybe it's not for a user like me, who would probably use it twice a month, if that. But I was interested in checking it out but got immediately priced out. For a startup, it seems like an ill conceived practice.
But what do I know? I've only been using and paying for software for 30 years or so.