Httpx: A Ruby HTTP library | Hacker News Reader

Httpx: A Ruby HTTP library

gitlab.com

157 pointsdamir58 comments

Never understood why people prefer APIs like this:

  page1, page2, ... = HTTPX.get("/page1", "/page2", ...)

Why encode parallelism into a single polymorphic function? Why not use composition instead? Now you're stuck with this API forever. And if you want to customize parallel fetches (eg: batching) you'll need another different looking API.

Hi, I maintain httpx. I'll try to answer your concerns.

The API was not a question of preference. If you look at most http lib APIs (python requests included), a single call returns a response object. My goal was to keep compatibility with this API to ease the migration effort, while baking in support for concurrent requests, which was the whole point of the lib.

There is no parallelism in that call btw. It's in the Readme. If it's http/2, it'll multiplex requests. If it's http/1, it'll try to pipeline, and if doesn't work, it'll fetch them one by one (with or without keep-alive). If they're different origins, the requests will be managed inside an event loop.

> Why not use composition instead?

You'll have to explain with an example of your own. However, there's already a ticket for implementing an event-based request handling, I just never prioritized it because no one asked for it yet, no one has shown interest in contributing, and frankly, I haven't needed it yet.

> Now you're stuck with this API forever.

It's still early 0.x days, so I might still remove APIs I don't think are future proof. That said, a lot of libs have lived with similar APIs all these years, and I don't see any reason for removing it.

Hey, very impressed with httpx. I'm not in a spot to contribute code but would like to sponsor (if that's an option). My email's in my HN profile and apologies for dropping this into the thread I couldn't find contact info elsewhere.

Thx for the kind words. I thank, but will not accept financial contributions from individuals. Feel free to use it, modify it, or talk about it to your colleagues :)

Nice lib, eager to use it in replacement of HTTP.rb, auto managed persistent connection is a great addition.

Do you plan to support webmock and httplog ? Or add support for httpx into those ?

Can't integrate a networking lib that can't be mocked in my specs.

Thx! I don't have any plans to integrate them, as I don't use any of them. But it should be trivial to create a plugin for them using the httpx plugin system: https://honeyryderchuck.gitlab.io/httpx/wiki/Custom-Plugins , feel free to give it a try.

httpx also ships with its own request/response logger, set HTTPX_DEBUG=1 or set the options so that you see an example.

Superb syntactical sugar.

> It's still early 0.x days

Any idea why the project is posted now? Is there a beta or RC release that’s significant?

My guess is that someone found it interesting and worth sharing with the HN community.

Do you mean, that vs something like this?

    responses = HTTPX.get("/page1").
                      get("/page2").
                      start

My guess is that the focus of HTTPX being parallel requests, the author choose an API that explicitly groups them together. The documentation is pretty sparse, I wonder if we can pass an array of URLs to get().

yes, ruby has a splat * operator that will basically expand an array into a set of arguments.

    array = ["https://news.ycombinator.com/news", "https://news.ycombinator.com/news?p=2", "https://news.ycombinator.com/news?p=3"]

    page1, page2, page3 = HTTPX.get(*array)

Note you can also splat the return as well:

    *pages = HTTPX.get(*urls)

splat?

   urls = ['https://example.com/foo', 'https://example.com/bar']
   HTTPX.get(*urls)

And I guess it won't return unless all pages are fetched? even worse...

There are timeouts involved, that is, you'll get error responses. Feel free to try the example scripts.

> ...while baking in support for concurrent requests...

Timeout does not help here - if the api is for concurrent requests, it's bad that caller cannot do anything before `(the slowest request finished || timeout exceeded)`.

We are using http.rb at https://serpapi.com

It’s interesting to see that the Ruby ecosystem doesn’t seem to find its “requests” library. There is so many very good competitive libraries: native net/http, HttpClient by Nahi, http.rb, HttpParty, RestHttp, RestClient, excon, em-http, faraday, curb, etc., and now Httpx!

Don't forget Typhoeus. That's literally my Goto "nuke the API with 10,000 concurrent requests" library.

It's also based on Curl so very stable.

https://github.com/typhoeus/typhoeus

    hydra = Typhoeus::Hydra.new
    10.times do
      request = Typhoeus::Request.new("www.example.com", followlocation: true)
      request.on_complete do |response|
        #do_something_with response
      end
      hydra.queue(request)
    end
    hydra.run

True. Typhoeus is awesome as well.

Its almost emblematic of Ruby's mindset vs. Python's "there is one right way to do something" philosophy.

Except for the language itself: Python 2.x vs 3.x are still battling choices after 10 years :)

Ruby had the same problem for a while, and only got over it more easily because of lower volume and diversity of legacy code (because it had far less penetration hang Python and almost all of that in Rails at the time of the 1.8/1.9 split.)

There is no battle. Python 2 updates have been unsupported for 10 years, and full support dies with this year.

It is a common misconception, but that is not the Python philosophy.

I've been using httparty, since forever.

Am I right to assume the primary difference with this library is concurrent connections?

https://github.com/jnunemaker/httparty

httparty is a dsl wrapper around net/http, so it's also limited by it. For instance, it doesn't come with keep-alive support (I'm using httparty in my dayjob legacy code).

httpx is more than concurrent connections. It's actually about concurrent requests done right, advanced http features (it can do connection coalescing, alt-svc), and it comes with a baked-in plugin system which makes it easy to extend.

Go through the wiki and give it a try when you can.

Me too but I am intrrigued.

Hmmm... https://www.encode.io/httpx/

That one seems more recent. Would be cool to know if ruby's httpx was the inspiration behind it (I can only judge looking at the name :) ).

Python HTTPX co-maintainer here: AFAIK, Ruby's HTTPX did not inspire Python's HTTPX.

Eg. the name came from a totally unrelated discussion, and research did not yield results about Ruby's counterpart back then [0].

In fact, Python HTTPX started with requests-async [1], which was then spun up into httpcore, which was then renamed to http3, and finally HTTPX.

Still very funny to me that these two projects (Python HTTPX and Ruby HTTPX) have so much in common: HTTP/2 support, API compatibility with the current de-facto HTTP library of the language, etc.

[0]: https://github.com/python-http/discussions/issues/1#issuecom...

[1]: https://github.com/encode/requests-async

LOL I was almost posted a ref to HTTPX until I saw this thread :p Thank you for HTTPX. I only learnt about it recently, but it’s now my go-to tool moving forward. Excellent work!

True, there are indeed a lot of common goals, and even features. Even our debug flag (HTTPX_DEBUG) is the same. Is that a python pattern?

Good luck in our work!

It's astounding to me that so many platforms/languages still have such mediocre http libraries. Surely basic http capabilities should be built-in or in standard libraries?

It is part of core Ruby. https://ruby-doc.org/stdlib-2.6.5/libdoc/net/http/rdoc/Net/H...

People develop other HTTP packages usually for ergonomic reasons, but the HTTP from core Ruby works fine.

Also, rubyweekly #474 [1] has HTTPX gem listed on top of the issue. Congrats to HTTPX author on well deserved attention.

1: https://rubyweekly.com/issues/474

This looks interesting, thanks for sharing. I've always had good success with httprb[1], but I'll give this a look over too.

[1] https://github.com/httprb/http

Does anyone have any plans/thoughts/visions for how we should deal with SRV-records in our standard libraries in different languages?

Requesting a DNS-server for a SRV-record gives you a prioritised list of IPs and ports, and it is up to the implementor to pick the right one.

As it works currently you would have to do a lookup, pick a endpoint, and then feed it into your HTTP library or TCP library.

This is fine in code that you control, but there are so many libraries out there that expects a hostname or an IP-address with a port number, and in a highly containerised world endpoints are more short lived than ever, so we need retry mechanics as well: Try the next server:port pair in the prioritized list.

Great library!

Although choice and experimentation certainly has its value, I'm curious to hear if there was any attempt to build the missing features into any of the existing libraries? (for example excon or http.rb which this is based on)

It wasn't until I read a comment here that there was a wiki that I found most of your documentation. I'd link this higher in your README as it is what I was looking for when I went to the project (lots of examples):

https://gitlab.com/honeyryderchuck/httpx/-/wiki_pages/home

I found this gem (pun intended) only after having an issue with original awesome http.rb gem not being able to talk http2. Happy httpx user ever since. Ping me if you decide to start accepting donations/support payments.

This looks like a pretty awesome library and very well-written and documented library. I'll definitely try it out in some of my Ruby projects and I would love to see an adapter for Faraday.

Kudos to the author(s).

Thank you. And there's already one :)

Why would I choose this over Faraday?

Faraday is a wrapper around lower-level libraries like this one.

Faraday would likely need some modification to support the concurrency aspect, but eventually you could be using Faraday _with_ httpx, rather than instead of.

Good to know, thanks!

Random comment - I find it interesting that I have such a strong association of open source = github, so when something is on gitlab, I automatically deduct mental points, even though that's not fair at all.

I think gitlab's sidebar is probably not helping with my association, it's too similar to bitbucket or other "enterprise-y" type websites.

Personally I find the UI and terminology so willfully - almost obstinately - different to github's, that it's an absolute pain to navigate. I'm sure I could get used to it, but just now I just wanted to jump to the code, and it took roughly 30 seconds to find it hidden inside the 'repository' tab.

Yeah, some of it is just familiarity but I do think GH truly nailed UI to quickly evaluate a project while also being good for working in long-term, at least on desktop—I have complaints about their mobile UI, but when I'm using that it's usually because I'm just curious and messing around anyway, so it's not a big deal. It's also nice and snappy.

[EDIT] I also assume a higher likelihood that key persons on the project are the contrarian-for-contrary's-sake sort if they're not on GH. Not that it's certain, mind, but that it's a bit more likely.

I don't think your issues with Gitlab's UI are the UI's fault. In addition to clicking on the "repository" link (which is a common term used by both Git and Github themselves) or scrolling up as others have noted, you can also click on any of the first three links immediately under the gitlab logo ("httpx", "project", and "details") to get to a view similar to github's default.

The link is already in a page similar to the main page of repo on Github. Open and link and scroll up?

Ugh, it's a link to the `README`, with a fixed sidebar. I partially take back my original comment, but I'd still argue that it's not the most helpful UI.

How can we make it more helpful?

Maybe I've spent enough time with GitLab, but I find GitHub's UI annoying.

Not browsing the repository / looking at a file? Can't go to the "jump to file" page.

Not at the root of a repository? Can't access releases.

Their "mobile UI" (which, I think, is being rehauled?)

Even ignoring any possible trust issues with Microsoft's ownership of Github, I think the tech world's reliance on a single host is rather dangerous. For instance, there's a story from today about the Spanish gov't ordering Github to remove a repo[1]. And there were technical issues for Chef when someone removed their code in protest of ICE[2]. Git can be decentralized, and that's one of the good things about it, but we the community (myself included) are not decentralizing very much. Github isn't really to blame, since they made a pretty great host and made so many things easy to do with it. It's in their best interest to get every using it. But it's not necessarily in the community's best interest to be relying on it to the extent that we do, to the extent where people deduct mental points for non-Github projects. I say this having done such mental deductions myself in the past, but now I think alternatives to Github are necessary and should be celebrated. Don't necessarily stop using Github, but if you keep using Github, mirror the code to at least one more host, be it Gitlab or elsewhere.

Or maybe it's just inevitable that what was once distributed always becomes centralized again? (To that end I'm reminded of the book The Master Switch by Tim Wu. He discusses similar cycles happening to various "information empires".)

[1]: https://news.ycombinator.com/item?id=21395629 [2]: https://news.ycombinator.com/item?id=21025037

Very good point. I'm so annoyed by the tech world's tendency to fall into the "winner take all" phenomenon that happens, and how this gradually chips away at the rights of users in very subtle ways. Case in point is Chrome, and how much power it has over the direction of the web now - leading towards a walled garden web.

httpx maintainer here. I chose gitlab at a time where I used the community edition inside the company I worked in. We had our onprem infrastructure and needed some private git management dashboard a-la github, and gitlab (even requiring gobs of RAM) saved us.

I also decided to close-source this project initially, as I didn't know where it was headed, and github didn't provide private repositories at the time.

It also provided CI out-of-the-box, whereas if I had chosen github, I'd have to integrate it with travis, or circle-ci, or yet another third-party that required me to update my configs every 6 months. I can tell you only this feature makes the decision totally worth it.

Also, gitlab has way more dev-friendly features than github, which has been recently only copying what gitlab already does.

I do mirror the project to github, in case anyone from there wants to contribute, so I'm not that narrowminded. However, I do prefer gitlab, and it's where I do most of my personal work nowadays.

I do the same as well.

My brain: GitHub or bust.

Thanks for the feedback! The UX Research team has an open Epic for outlining sidebar improvements and it'd be great to hear any suggestions you may have. Feel free to leave a comment in https://gitlab.com/groups/gitlab-org/-/epics/577 or share additional thoughts here!

Reply on news.ycombinator.com