arcatek

> Which lightweight shell will be used on Windows? Does it also bundle standard unix tools (if a script pipes to grep or less for example)?

It will be in-house, and very basic. We don't intend to rewrite bash, just to provide the basic experience that is usually needed when adding script into the `scripts` field. For more complex needs we'll simply offer a way to opt-out and use the native shell, or to call Node scripts.

> How will paths be translated on Windows?

The current Yarn tries to do this by using the `path` native module. It's quite error-prone since backslashes tend to appear in the worst possible places. For the v2 I plan to work with all paths in a posix style, and convert them into Windows paths right before they reach the filesystem (which is similar to what Cygwin does, as you mentioned). It would be a bit slower on Windows, but massively simpler in the codebase.

Most of those have been fixed a long time ago, but we simply don't have the resources to triage them.

This effort we start is in no small part to decrease the number of issues that will be created by empowering the users to unblock themselves* and solidifying Yarn's codebase.

* You wouldn't believe the number of issues that are simply about things working as they should - we can't really blame their authors because it can be quite hard to find the right paragraph in the documentation, but it's extremely taxing on a small team. Similarly, we often have issues created against older releases, or without reproducible test case.

We want the lockfile to be easy to review by humans. In our experience, JSON doesn't quite fit the bill once you reach a critical mass of data.

Our lockfile format worked fine for the past three years (bar the unfortunate YAML incompatibilities that we're about to fix). Don't fix what isn't broken :)

We'll make sure to add a notice at runtime to make it clear (plus, a consequent diff at review).

Also note that we recommend using `yarn policies set-version` to enforce the version of Yarn used by everyone in your team with very little friction:

https://yarnpkg.com/en/docs/cli/policies#toc-policies-set-ve...

Worth noting that the principle of "a package can only access the dependencies it declared" is already something that we (Yarn) are pushing through Plug'n'Play.

We're not focused on security (yet), but any help we can get to move the ecosystem towards a stricter model will help you in the long term (by ensuring that common tools will be compatible with the even stricter model you're advocating).

[1] https://github.com/yarnpkg/rfcs/pull/101

I meant "at your own pace" as in "they let you decide how you want to learn it". Meaning that instead of following classes (which were quite basic anyway) I spent most of my time reading man pages and reading obscure forums, and they were perfectly fine with that.

Those who had issues were those who needed to be told where and what exactly to look for. They often felt like they were paying for little to no teaching (which is true, in a sense, but I preferred to see it as an opportunity to grow by myself in an environment where I could have extra resources at my disposal if I needed to - the best one often being other students, sometimes from previous years).

Something interesting is that the way Epitech (and 42) work is completely different from what you would have in public schools (at least in France).

For example, one key part for me was that the school allowed me to practically don't go to classes as long as I could prove that my work wasn't affected. It's something that you won't find everyday. I'm 100% sure I'd have failed university otherwise because the lack of freedom would have bored me to death Conversely, friends of mine have dropped from Epitech in the first two years because they didn't get enough support, so it shows that the most interesting thing the school has to offer is its learning devices. They understood that one size doesn't fit all (which is against the French public school philosophy).

So yeah, the tuition is expensive (and frankly given how the school works you sometimes wonder whether they eat luxury cars for lunch), but there's simply no public alternative.

One other factor I've also wondered is if people paying for their scholarships weren't more invested in making it a success? I knew my fair share of people who weren't thriving as developers at Epitech as well, mind you, but most of them dropped before the end.

It doesn't have a lot of advantages right now. That said, it won't stay true forever. PnP is but the first step of a long-term goal I have. Check the Section 6.B from the whitepaper[1] for more details.

[1] https://github.com/yarnpkg/rfcs/blob/65b36475c04b1149eb51a81...

Flow is already 100% compatible with PnP, through the use of the `module.resolver` configuration settings. We even saw some slight perf increases after switching in.

I guess the same could be done for other tools: the .pnp.js file can be easily interfaced with anything you throw at it without them having to care about the way the dependencies are resolved under the hood. Even non-js tools can simply communicate with it through the cli interface, which returns JSON data ready to use.

Since the resolution is new, most of them requires plugins. I already wrote those for the most common projects:

https://github.com/yarnpkg/pnp-sample-app/tree/master/script...

They'll be published as separate packages once we merge the PR.

We'll see what the community response is, and once we're confident this is what everyone wants we'll merge it into master (PR is already up[1]).

While there isn't an official build at the moment, the code PR includes a prebuilt version of the current branch, and we have a playground repository to experiment with it.

[1] https://github.com/yarnpkg/yarn/pull/6382

[2] https://github.com/yarnpkg/pnp-sample-app

> what's being done to make this the default behavior in node and deprecate node_modules proper

My hope is that PnP proves that this approach can work and raises the stakes for Node to implement APIs that would allow for a better integration. It's a long term plan and we have yet to prove ourselves on the long run, but I'm quite confident :)

> If you remove support for them

I don't think we'll ever remove support for them. I'll personally advocate for them not to be used because of their subpar developer experience, but apart from that it won't cost us much to support them.

> when the module you're looking isn't in the static resolutions table

The fallback will kick in when the package that makes the request isn't in the static resolution table. Since those packages aren't part of the dependency tree we assume their dependencies have been installed independently, hence the fallback.

That said, I think the use case described by the parent post is neatly solved by the `link:` protocol, which basically 'aliases' a module to a specific name.

The .pnp.js file can be checked-in, but needs the cache to work. Right now I'd advise not to check-it in.

`yarn unplug` will eject a dependency from the cache and put it into the project folder. That's a quick and easy way to inspect and debug libraries you use.

I did work on the asmjs/wasm bindings for Yoga, Text-Buffer, and a few other projects, so I'm biased :)

Anyway, regardless of my own long term feelings, rest assured that postinstall scripts will be supported as well as they are now.

Yes, we only do a single stat in those cases (to check whether it's a directory or not).

Someone working on the wasm will likely be able to tell you more, but from what I understood dynamic linking is on the table[1]. I didn't quite say that it was ready yet, but rather that it was on the right path.

[1] https://webassembly.org/docs/dynamic-linking/

I mentioned it in another answer, but we'll eject the packages that require postinstall scripts - so they will work as before, but will take extra time to be installed.

As for wasm, I'm curious to hear what you think isn't good enough. I think the two main issues are garbage collection and dynamic linking, and there's ongoing work on them to fix them.

It works in two steps: the first step resolve to "unqualified paths" (which are just the paths within the cache without the index.js/extensions resolution). This step is entirely static, no filesystem involved here.

The second step converts the "unqualified paths" into "qualified paths", and is basically the index.js/extensions resolution. We currently access the filesystem in order to resolve them (just like Node), because we didn't want to store the whole list of files within our static tables (fearing that it would become too large and would slow down the startup time because of the parsing cost).

So to answer your question: we get rid of the node_modules folder-by-folder traversal, but decided that the extension check was an acceptable tradeoff. We might improve that a bit by storing the "main" entries within the tables, though, which would be a nice fast path for most package requires.

I think you're referring to the package-name-maps[1] proposal. We've been made aware of it during the middle of the development, and while we chose to continue using the static data tables approach, they aren't incompatible.

The reason we abstracted the static tables behind a JS api is precisely to make it easier for everyone to experiment with various implementations - some might use the internally embedded data as we do, some could consume a package-name-maps file, and some could do something entirely different. As long as the contract described in the document is fulfilled, everything is possible.

[1] https://github.com/domenic/package-name-maps

Postinstall scripts are the main issue, yep. Right now the current implementation doesn't do anything special with them, meaning that they are installed inside the cache (except when they're disabled altogether, which often works well enough since there isn't that many packages that require postinstall scripts).

This is obviously wrong, so we'll soon go to a model where we "unplug" the packages and put them into a specific directory (a bit like the node_modules folder, but entirely flat). The feature itself is already there (`yarn unplug` in the rfc), but we need to wire it to the install process.

Ideally, I think it would be beneficial for the ecosystem to move away as much as possible from postinstall scripts - WebAssembly became a very good solution for native libraries, and as the added benefit that it works everywhere, including the web.

Workspaces are working just fine: instead of creating the symlinks we simply register them into the static tables, and resolve them from their actual location on the disk.

Nohoist was a bit of a hack from the beginning (precisely because the hoisting was anything but guaranteed), so some incompatibility might happen for packages that cannot work without.

That said, I'm not too worried: we've tried a bunch of the most popular open-source packages, and they all seemed to work fine - the one issue was on an interaction between create-react-app and eslint, but there's discussions going on to fix that in a clean way.

I'm the author behind the proposal, feel free to ask me any question you might have!

As a personal note, I'm super excited and so grateful to have had the chance to work on this project - node_modules have been a longstanding thorn in the side of the Javascript ecosystem, and to finally have a chance to try something completely new is amazing.

I work on Yarn. The message you're quoting is from the npm CTO. We don't do yarn-specific changes. The main reason we're using a mirror is because it was massively faster than directly querying the npm registry when Yarn got released. The difference decreased during the past year I think, but Cloudflare was still a bit ahead in most scenarios.

Yes, the gc isn't exposed yet, so neither JS nor wasm are notified when an object goes out of scope (which would allow to reclaim the memory on the wasm side). It's on the roadmap, so it should eventually come.

For the time being you should be able to install it using the `--devel` flag, but the PR to move it to stable is on its way :)

We've also published a more detailed documentation here: https://yarnpkg.com/en/docs/workspaces!

I unfortunately don't remember - possibly a combo of Babel + Webpack and their plugins, tho, they're usually my go-to choices to test install perfs.

Now that I think about it, it's quite possible that the process was apparently hanging on the same issue I describe in my article, where babel-core depends and babel-cli and vice versa. Maybe it would work better if I were to run the process a first time with a simple algorithm like the one exposed in the article, that would clear up any dependency loop, then a second more complex pass that wouldn't have to deal with this.

I've made some early work on using constraints when I started working on Yarn, but it ended up unpractical because of the sheer execution time it required. A huge number of constraints were required to satisfy the requirements, which was made even worse when you consider that a dependency version might have different sub-dependencies than its other versions.

That being said, I'm not an expert in SAT logic, and it's quite possible a better solution is possible! Maybe some kind of hybrid algorithm, possibly? In fact, one thing I'd really like to do in medium term would be to externalize the resolver part of Yarn - this way, it would be much easier to experiment with it to try to find algorithms that fits various requirements. If you want correctness you would use a slow but comprehensive algorithm, if you want speed you would use a naive algorithm like the one in the article, etc.

Author here! Our goal in this article wasn't to build another package manager (we've got our hands already full with Yarn :), but rather to give JavaScript developers some insight as to what's going on under the hood every time they install their dependencies.

We hope it will contribute to demystify package managers, and give people plenty of ideas to improve their favorite ones - Yarn is a community project, and we're always happy to see new contributors step in!