Request’s Past, Present and Future | Hacker News Reader

Request’s Past, Present and Future

github.com

173 pointsHatchedLake72172 comments

https://github.com/request/request/issues/3142#issuecomment-...

mikeal commented 6 days ago:

> Let’s just bump the major version when we deprecate. That way most people depending on the project won’t see this error until they try to upgrade to a new major, which means they are actively developing it and really should look for an alternative.

I don't think that's a good idea.

The problem is that most of replacements are of lower quality than request.

I just moved to request from axios about a week ago. Axios has multi-year persistent bugs around proxy support, modifying https agents, and unhandled promise exceptions. Probably due to low contributions (the code base is quite complex). You only find these out after investing into axios heavily.

To new users axios looks superficially as good as request (similar number of users, promises by design, etc)

So... Semver? That seems like a great idea.

request will stop considering breaking changes.

Does the JS community seem to consider that a bad thing...? To me, this sounds more like "is becoming stable" than "is being deprecated".

“requests” doesn’t support (or apparently want to support) Promises which are now widely used in the JavaScript community, thus it’s obsolete. Promises work best when all the async operations in your code use them, especially with async/await.

Interestingly early versions of Node had a limited form of Promises but they were quickly removed.

Just to give a little bit more context, in contemporary javascscript you can write something like this (wrapped in a try-catch:

    async function http_request_new(url) { return new Promise(...) }
    const http_response = await http_request_new("https://www.google.com")
    ...

The await signals that the asynchronous http_request function is non-blocking, and we should continue execution once the non-blocking return value (promise) has resolved. Previously idiomatic javascript would have looked like this:

    function http_request_old(url, cb) { return ... }
    http_request_old("https://www.google.com", function(error, http_response) { ... })

Luckily node.js provides util.promisify that converts any old callback-style async function into one that returns a promise, so that await can be used:

    http_request_new = util.promisify(http_request_old)
    const http_response = await http_request_new("https://www.google.com")
    ...

Unless I'm mistaken, it's fairly trivial to promisify the old-style code

In the case of Request, there is already an official wrapper for the library that supports promises. https://github.com/request/promise-core

It is a couple hundred lines of code though.

Exactly - for the longest time, the success / error callback signature was the primary and expected way to handle async code in Node modules, and there was a lot of objection to using promises - in part because at the time it was not stable, and in part because its performance was less compared to callback style.

You don't want to put performance bottlenecks in core and/or frequently used libraries just so it fits your preferred async style. If you REALLY want to pay that price, it's trivial to promisify a library like this - and it doesn't add complexity or size or (worst case) dependencies (like 3rd party promise libraries, as was the case for a long time) to your small, focused library.

i guess these are the downsides of a platform without much stdlib. consensus stdlib doesn't really work, and then things don't work together.

util.promisify won't work on request in most places because they don't use the defacto standards of callbacks (the first argument being an error argument, callback being the last argument, etc...). Funnily enough, i believe it predates the widespread use of that convention! But that means you need to either do it yourself (which admittedly is pretty trivial), or use custom bindings to "promisify" request (which do exist! and are even maintained by the original authors).

But promises are only one reason why request is more difficult to use.

Compare the node.js style streams with async iteration which could be possible if the library supported it:

node streams style:

      const readStream = fs.createReadStream(inputFilePath, { encoding: 'utf8', highWaterMark: 1024 });
      readStream.on('data', (chunk) => {
        console.log('>>> '+chunk);
      });
      readStream.on('end', () => {
        console.log('### DONE ###');
      });

async iteration streams:

    for await (const chunk of fs.createReadStream(inputFilePath, { encoding: 'utf8', highWaterMark: 1024 });) {
      console.log('>>> ' + chunk)
    }
    console.log('### DONE ###')

Not only is the latter easier to quickly grep and understand, but it also is easier to catch exceptions (a try/catch works, and it bubbles up! So it's harder to ignore errors by forgetting to add a `readStream.on('error'...)`), and it works correctly in async contexts so it doesn't also need to be wrapped by Promise constructors.

Then throw in writeable streams, transform streams, and tons more that all require more difficult setup, less standard ways of working, and are overall harder to get completely right without any bugs.

With async/await you have to wrap everything in try/catch, so it only looks better if you do not handle errors. It also makes it harder to do more advanced designs like rate limiting, loading bars, etc. I really like callback convention with error first. Something being async also means it can fail. The most important part of async code is error handling.

errors bubble in async contexts still, so this example will still catch errors correctly:

    async function foo () {
      throw new Error ('test')
    }
    
    async function bar () {
      return foo()
    }
    
    async function baz () {
      try {
        await bar()
      } catch (err) {
        console.error(err.message)
      }
    }

That's not the case in callback-based systems where it's more of a golang style "handle it right then, or it gets ignored forever" kind of thing.

But also unhandled promise rejection handlers are seeing much more widespread usage now to catch "unhandled promise rejections", which make it a bit easier to handle errors in async contexts, but i'll be the first to agree that there is still a lot that sucks about handling errors in async contexts in javascript...

> It also makes it harder to do more advanced designs like rate limiting, loading bars, etc.

It does, but you can also handle a lot of these by mixing promises and async/await (since a/a is basically a nicer syntax for Promises). And callbacks still exist for the cases that really need them (like you said, loading callbacks are still a hell of a lot easier to use than janky promise-based solutions in most cases in my opinion! Mostly because a promise is one-and-done, but callbacks can be re-called multiple times).

But the point is that if you stick to ONLY callbacks, EVERYTHING has to handle wrapping it. Whereas if you move on to using the new features where they make sense (the "easy path" can use async/await or async iterator streams), then only people who need those extra complicated features have to wrap and work with those more complicated setups.

I think there are use cases where Promise and async/await makes sense, for example transactional SQL, so you can break up the transaction into many simple queries instead of one huge complex query and without the added boilerplate from the callback pattern. But when you add Promises, they tend to spread. A middle ground is to have both Promise and callback, eg. if(!cb) return promise. Another gripe with Promise's is that "soft" errors are treated the same as catastrophic errors.

While I agree that there is room for both (callbacks make sense in some areas, promises/async/await in others). I feel that promise-based async should be the default, since it provides so many additional benefits.

>A middle ground is to have both Promise and callback, eg. if(!cb) return promise.

This works in many cases, but becomes ugly in some. The 2 ways of working have subtle differences (callbacks and promises both trigger at different times, and there is some interplay with "microtasks" and queues and schedules and other things that make some situations hairy). There's also the problem of Promises being hard to use when additional closures are involved (it's why `.forEach` sees much less usage in async/await codebases, because it's a lot more cumbersome to "wait" on iterating over an entire array with `.forEach` because the inner `await` can't escape the function in `.forEach`.) So most users will end up choosing one or the other for the whole library, and at that point why not just have them be 2 different libraries?

It also makes documentation harder, and makes the implementation of the library a lot more complex (now it needs to do EVERYTHING in both a promise-friendly way, and a callback-friendly way), and some things just can't be easily split (how do you support both async iterators and node.js event streams at the same time in the same interface?)

One strategy is to be adapter friendly by sticking to conventions. By adapter friendly i mean promisify and event emitter to async iterator adapter. And vice versa. And by conventions i mean things like error first. Then it might be possible to just have a .promises property that expose the api via the adapter. So that you can support many paradigms, with minimal changes to the code base. With paradigms I mean different ways to work with async, there are probably 20 different ways that I know of if I also include ways how to run things in parallel. My personal opinion is that promises should have stayed in userland (not in ES spec) to give competing standards a chance to compete. Those wanting async/await patterns could have used co routines/generators.

When you add asynchronous operations, whether they use promises or callbacks, they tend to spread beacause any consumer of an async operation probably needs to be async itself. That’s not specific to promises.

If you return a promise you don’t really need to support a callback because you can just call .then on the promise with a callback. i.e.

    func().then(result => {
      // success
    }, err => {
      / error
    })

Is equivalent to

    func((err, result) => {
      if (err) {
        // error
      } else {
        // success
      }
    })

IMHO promise-based APIs are strictly better than callback/“errback” style APIs for many reasons, async/await support in the language being the nail in the coffin.

The idea of exceptions is to catch them where you would handle them. If your code is catching to re-throw, catching to add a header and then rethrow, or catching to revert back to rcode-style error handling, then you aren't using try/catch to its most practical potential.

With this philosophy in mind I've never been overburdened by try/catch littering my code.

> With async/await you have to wrap everything in try/catch, so it only looks better if you do not handle errors.

Better than having

    if (err) {
      callback(err);
      return;
    }

at the top of every function.

> It also makes it harder to do more advanced designs like rate limiting, loading bars, etc.

No it doesn't. Rate limiting is no harder with Promises or async/await. Loading bars can be handled with async iterators. At the worst, one only needs to fall back to callbacks in such situations, and abstract that in a Promise wrapper for use with async/await.

> I really like callback convention with error first.

It has completely different semantics from the rest of the language. You must handle an error wherever it's thrown, which is ridiculous.

> The most important part of async code is error handling.

Sure, and that handling should be done at the highest level possible.

> Unless I'm mistaken, it's fairly trivial to promisify the old-style code

Even if it was trivial to promisify request (though it seems it isn't according to sibling comments), the point is that the promisify utility is effectively a workaround for working with older libs & APIs in a modern idiomatic way. It's supposed to be a temporary shim to move into modern patterns, not something whereby you start a new project, choose new actively-developed lib, and expect to have to wrap that lib just to use it idiomatically.

Similarly, "upgrading" your popular lib to have a modern API just by wrapping it with the promisify util and leaving the internals using a legacy pattern not only adds overhead, but it also just a bit of a hack, and not very ideal. Especially when more modern alternatives to your lib exist which you could be recommending.

There is an [official] additional package "request-promise" that adds Promise support. I don't dabble in Node much, but is this a bad thing? Although on the other hand, most other libraries support promises out of the box (e.g. using the "pass no callback to get a Promise back" pattern).

There's also axios, superagent, and a whole bunch of excellent node HTTP clients (also 'fetch' if you like encoding URLs and parsing JSON manually).

Bolting on the newer features as additional packages has a few downsides in my experience:

* Increased complexity. So now in order to use request in a modern JS ecosystem, you need to install request, and request-promise-native, which itself installs request-promise-core, which all 3 then combine to use. Which do you look for to find documentation? Which do you look at when you have bugs? How does interop happen with other 3rd party libraries? Does every other lib that works with request have to handle all the official wrappers or just the main? And good god what happens when breaking changes are made to one of them? Everything is just a lot more complicated here.

* Only so much can be easily patched on. Things like streams are pretty complicated, and writing a complicated (and often slow) runtime transformation of a stream to an async iterator, both of which don't have complete 100% matches for all of their features, means that everything is now using a sub-par incomplete interface. The main application can't implement features which take advantage of the new syntax, and the wrapper can't always represent the old ways of working perfectly.

Basically, it's a leaky abstraction in most cases, which is slower, uses more memory, and is a lot more complex.

I use request-promise and it’s always worked perfectly fine. I normally do

  const response = await rp({
    request params
  });

and then continue on.

No, but `request` does not implement modern JS features such as promises and async/await which generally have become commonplace. So what this announcement means is that such features won't ever be added, and that people should look to another library for HTTP requests in "modern JS".

As I replied to tlrobinson above, is this a problem when there is an an official package that bolts on a promise support?

As the post explains, Javascript has changed significantly since request was first created, and that left them with 3 real options:

1. Try to make massive breaking changes to the request module to make it more easy to work with in the current javascript landscape (promises, new streams, async generators, etc...). This is a problem because then the millions of blog posts, comments, stack overflow answers, and more are all now outdated and wrong. And as the author explained, it would basically be a completely new library.

2. ignore it and become a thorn in the side of devs everywhere (oh, i can use new streams and promises in 99% of my codebase, except for a few old modules that we still use or have to code around until they get removed as they are difficult to work with)

3. Deprecate request, and make a new module under a new name which can adapt to the current landscape.

So if you value non-breaking changes over developer experience and complexity of integration, then it's a good thing for you. If you value tools which work with the language as it exists now rather than against it, it's also a good thing for you since the new "request by a different name" will work much better in current javsascript.

This is the opposite of what Angular did. Angular 1 vs 2+ were basically different libraries. Rather than take the angular route and just increment the number and keep the name, they are trying something different. They are throwing away the name and starting fresh.

I'll start from bottom options:

3) "deprecate request and start new" - and discard 10 years of commits, stability and tests? Wouldn't be a rational way to go.

2) "ignore it and become a thorn in the side of devs everywhere" - it wouldn't become a thorn more than it is now, would it? It's a 10 year old package that's downloaded 14 million times a week. It doesn't feel a thorn with those numbers. You can also `util.promisify` it if a callback is a "thorn" for you as a developer. (but see https://news.ycombinator.com/item?id=19604867)

1) "there are a lot of google results with solutions that the new breaking changes will make obsolete" - that's what semver and major versions are for. "We'll break Stack Overflow answers" doesn't feel like a valid reason. jQuery has had its fair share of breaking changes in the last 12 years, and we know the numbers if we compare `jQuery` vs `request` questions on Google/Stack Overflow, so again, it doesn't feel like a valid reason.

> 3) "deprecate request and start new" - and discard 10 years of commits, stability and tests? Wouldn't be a rational way to go.

The new package doesn’t have to start from zero — it can start from where the old package left off. In terms of version control history it could even start life as a copy of the old git repo.

The developers having worked on the old version for that long certainly know what will be best for them I should hope. Whether that means reworking the old code for the new package, or taking some parts of it with them, or rewriting everything. As long as they don’t fall for the temptation of implementing new features in the rewrite, or at least not any significant number of new features.

> 2) "ignore it and become a thorn in the side of devs everywhere" - it wouldn't become a thorn more than it is now, would it? It's a 10 year old package that's downloaded 14 million times a week. It doesn't feel a thorn with those numbers. You can also `util.promisify` it if a callback is a "thorn" for you as a developer. (but see https://news.ycombinator.com/item?id=19604867)

Just because people are using it doesn’t automatically mean people enjoy using it.

As another commenter answered 3 and 2 (and I agree with them), i'll answer 1:

There's a difference between "has some breaking changes" and "will have basically an entirely new API, and will support different features and make different tradeoffs"

Like for instance AngularJs 1.x vs Angular 2+, or what seems like every major version of react-router, or various other examples of this that come back around and bite people. The only difference with those is that they don't have literally 10 years of documentation, blogs, SO answers, and more with no version numbers on them.

This isn't an instance where dogmatic application of the "rules" is a good idea. This is one where choosing to make a new library under a new name is the lesser of 2 evils. The hope is it will cause less problems, less confusion, less difficulty for both new and old devs, and be better for everyone.

I don't know about you, but i've absolutely spent ~30 minutes on multiple occasions trying to figure out why some documentation seems to be wrong, or a feature isn't working in a library only to realize that i'm not on the latest version or the docs don't have an easy way to go back through old versions, and it's a big timewaste as well as a source of bugs. This move is trying to prevent that!

> There's a difference between "has some breaking changes" and "will have basically an entirely new API, and will support different features and make different tradeoffs"

Doesn't this library also predate the semver spec, and wasn't just such major alterations the purpose of the major version number back then?

Wouldn't there be a 4th option of adding support for the newer patterns without breaking the old?

Hmm, so there are 41K packages depending on 'request' but from now on everything that is merged into master is immediately going to be published?

Sounds like a recipe for disaster not unlike what happened with event-stream.

At least they're trying to manage it responsibly:

> We’re going to have to remove inactive collaborators and enforce 2fa, because commit rights will effectively become npm publish rights.

"Let’s just bump the major version when we deprecate. That way most people depending on the project won’t see this error until they try to upgrade to a new major, which means they are actively developing it and really should look for an alternative."

You won't be able to enter the "recipe for disaster" without at least seeing the warning when you update your dependencies. At that point it's your prerogative if you want to continue with a deprecated package.

I think a comment in this thread should be visible to highlight any alternatives to request.

I've been mentions to got (https://www.npmjs.com/package/got), if anyone cares to point to other alternatives, that'd be nice, if only for educational purposes.

fetch() has an excellent API. It's native in browsers and it has a good Node implementation called node-fetch.

https://developer.mozilla.org/en-US/docs/Web/API/WindowOrWor...

I agree “fetch” is good enough in many cases. It doesn’t have all the conviences many clients do, but sometimes those can be added with a simple wrapper function tailored to your use case.

How come fetch hasn't been integrated directly into node, like how its integrated directly into browsers?

it uses a lot of infrastructure that node doesn't have, like Request objects and Response objects and Body objects and mime checking etc etc list goes on.

if someone feels like adding all those things to node and feels like arguing with our core maintainers about the duplication with http then maybe it will be added.

https://github.com/nodejs/node/issues/19393

Because Node.js already provides low level network access, while in browsers fetch() is the (second) lowest abstraction you can get to.

I don't think fetch provides progress indicators yet which can hamper its usage if you want good UX for certain operations like uploads.

Mikeal is working on `bent` [1] but I'm not sure I like how it results in a few lines extra over a `request` like library. Can anyone explain why this syntax?

[1] https://github.com/mikeal/bent

Axios (https://www.npmjs.com/package/axios) is probably one of the most popular.

Node really doesn't pay attention to nice API design. If all readable streams had

`Stream<T>.toBufferAsync(): Promise<Buffer>`

`Stream<T>.toStringAsync(): Promise<string>`

`Stream<T>.toArrayAsync():Promise<T[]>`

then you could quickly collect any stream you want into a promise, including the stream returned by request.

Here is how you do a Stream API properly.

https://api.dartlang.org/stable/2.2.0/dart-async/Stream-clas...

Has first, last, toList(), map, reduce, filter, takeWhile, skipWhile, Stream.periodic and other great goodies that make stream usage nice and easy.

Too bad Google never invested this amazing API designer power into JS, they could've been the "apache commons" of node.

We're getting there. I made both node and web streams into async iterables, and have now proposed this for js standardisation: https://github.com/tc39/proposal-iterator-helpers

I use this package... which mirrors the whatwg fetch API and has support for promises baked in:

https://www.npmjs.com/package/isomorphic-fetch

What's the alternative to Request? I see Mikeal has made Bent [1], but I'm not sure if the new, alien syntax is worth it? Also found got [2], thoughts?

[1] https://github.com/mikeal/bent [2] https://github.com/sindresorhus/got

bent seems too "bare-bones" for my daily usage. If you like that good for you, but I have been using axios[0] which is more heavy weight.

https://github.com/axios/axios

Problem with Axios is that it has almost 500 open issues and not enough maintainers.

We've moved to fetch, which is not as handy, but is the standard lib in the browser. Actually we use wretch which adds a more convenient API on top of fetch.

https://github.com/elbywan/wretch

For Node we use node-fetch which mimics the fetch API using the native HTTP module in the background:

https://github.com/bitinn/node-fetch

I'm using axios in production for over 2 years now. Can you point me to a certain issue you had I'd need to anticipate (or check if I already have)?

I've used it in production for 3-4 years with no issues (that I know of) but it's a ticking time bomb.

https://github.com/axios/axios/issues/1965

https://www.reddit.com/r/javascript/comments/an94xq/axios_ne...

How is it a ticking time bomb if all of the tests pass in that package?

I don't know much about the javascript ecosystem, but isn't this what major version updates are for?

There's precedent for outright deprecating a package when others exist that do a better job of meeting current development needs. With Promises and node/browser cross compatibility being the biggest ecosystem shifts, there are a few newer choices that already meet those needs. Also, `request` in particular would be gross to upgrade since so many people rely on additional dependencies that add compatibility for things like Promises, retry strategies, etc., all with varying degrees of active maintenance.

As someone who's tried to switch off of `request` recently, I'm worried about this. I didn't find any of the alternatives to be particularly viable.

Gosh. I was rather hoping the days of Javascript churn were over but reading this reminds me of why I lost interest in front-end web development.

`request` is mostly used on the server-side...

You still use xhr requests (or the easier fetch api) on the frontend. This is for the backend.

Most languages have multiple 3rd party libraries for HTTP requests, and I'm sure some a lot get deprecated.

Is the implication that every library module with async code must be rewritten to use promises or else be obsolete?

it's not just promises. as the maintainers of request continue trying to try to fix architectural bugs they are finding it would be easier to rewrite it from scratch rather than continue modifying the current one. this is said in about as many words in the linked issue too, so make sure you read before commenting ;)

"easier to rewrite from scratch" is a fallacy. https://www.joelonsoftware.com/2000/04/06/things-you-should-...

"Easier to fix if they allow themselves to completely break the public API" was how I interpreted the situation with request.

That's what MAJOR stands for in semver - when you make incompatible/breaking API changes

What's the reason for being in maintainance mode then ? I don't see it in the post though.

I have followed the twitter discussions and tl;dr is "the package needs a major rewrite and breaking changes for it to adopt Promises async/await and if anyone wants those there are more adapted projects already"

I because think maintenance mode is a responsible alternative to abandoning the package altogether as it has (from the post) ~41k packages that depend on it.

Clickbait title on HN? He said it's one of the first, not the very first one. Also just say "request". Anyone interested in an old npm package being deprecated will probably know this module. "request is being deprecated on NPM" is eye catching enough.

On top of this it's not being deprecated, although it seems they considered that option. It's just going into maintenance mode, which makes sense for a library which is essentially done (and why wouldn't it be, if it's been around for 10 years?).

https://news.ycombinator.com/item?id=19604560 explains why it is not really going in to maintenance mode though.

Ironically, I found the title to have the opposite effect; I read that "Node's oldest NPM package is being deprecated", and didn't think much of it; I imagine the oldest package in a repo is often just some kind of test package or some boring utility which isn't relevant anymore. It was first after clicking and reading that it's talking about requests that I thought, "oh, that's actually a kind of big deal".

We've reverted the title from “Node's oldest NPM package is being deprecated”.

Reply on news.ycombinator.com