“You meant to install ripgrep”
crates.io
crates.io
(I'm the author of ripgrep.)
Before you do something like that, always ask yourself: "What if everyone else started doing this?"
If the result feels like a nightmare in the making, don't do it.
Even flat namespaces are virtually infinite; a couple of extra names that correct user error do not pose a serious exhaustion risk.
Seems fine to me. Something like one tenth of packages reserving a second name? Not a big deal.
In CPAN, you create a module with a hierarchical name (Net::LDAP), and people inherit from it and extend the namespace to add new functionality (Net::LDAP::Batch). Finding a package that does what you want is [relatively] easy. Old code gets maintained rather than somebody reinventing it for the 72nd time with a hodge-podge of functionality.
No, I don't think so. There is no universality implied in my comment or in the specific practice here. You can make value judgments based on specific circumstances. For example:
* How many people try 'cargo install rg' and have it do the wrong thing? I'd say "probably a lot."
* Is 'rg' on its own something that is a likely useful or desirable name on its own? No, I don't think so.
This doesn't have to mean that everyone should do it for every possible alias of every crate out there. You can say things like "yeah I think it makes sense to squat a name here to improve failure modes for folks."
Other than that, I have squatted a few names before. I don't see anything wrong with the practice in and of itself. It's when it gets abused that it starts to become a problem.
1.) The use of names as a speculative financial instrument (in all shades of grey, up to and including extortion for lapsed or stolen names)
2.) The use of names as vectors of attack, such as by exploiting typos or homographs (such as malicious packages)
3.) The reserving of names you don't have a sincere or immediate intention to use (hoarding/FOMO)
This isn't very much like the situation with domains, which is primarily a result of #1 (there is no market for crates.io names, as far as I'm aware). #3 is a problem to some degree on crates.io, my understanding is that they basically treat this as a human moderation problem. #2 is endemic to all package managers.
By putting a helpful instead of malicious package here, the community (and Richard Dodd in particular) are able to mitigate the hazard of #2 (unless this account is compromised or turns malicious - a better but imperfect situation). If a project called `rg` comes around, they can appeal to moderators to get this name, and probably succeed (as if this were a #3 problem).
This isn't a perfect way to do things by any means, but it seems like a decent balance of concerns to me.
I think it's more accurate to say that they consider dealing with this out of scope. "I want this name that has been unused since it was added as a placeholder package 7 years ago" is not something that the human moderation will help you with. The extent of human moderation on crates.io is basically "This is malicious or illegal and was reported to us and we looked and agreed so removed it"
It is endemic to package managers which don't do curation, which is why I'm a fan of package managers that do.
EDIT: Found their email via git. Always forget about that one.
Yeah, but only when used together with fzf, the other favorite new cross-platform shell utility. I mean, after rg spits out a list you do want to narrow it down and then do something with the files in that list, right?
https://github.com/junegunn/fzf/blob/master/ADVANCED.md#ripg...
You’ve saved me tens if not hundreds of hours with ripgrep, and I’ve become a huge evangelist of it at my workplace. When I’m helping someone understand how to debug customer issues, the first thing I tell them is to install ripgrep. Truly a fantastic piece of software.
Amazing work on rg!
Truly the Unix philosophy at it's finest. It's the only way I search JSON these days! (or YAML with "yq | gron | rg" to get results, pop into (n)vim and to my thing :)
2. Auto ignores .gitignore files, so does not search node_modules or build/ etc., huge noise reduction there.
3. -t/-T gates on file extension which is a very nice feature, again signal to noise
4. The combo of speed and the above and the recursive-by-default mean that you search much larger corpuses by default, like “all the microservices in the cluster” or “my entire home dir”, because you know it's some .xml file mentioning “jackson” where you saw this config you need before.
5. For some reason I never remember which regex features are grep vs egrep, so I end up just testing on a bunch of strings to see if I have to like backslash the plus operator or whatever. With rg it's like “oh this is going to have the same syntax as JS regex.”
6. Unicode compatibility by default could save you that sort of time maybe on specific workloads?
I also squatted `memap` and `memap2` for the same reasons.
I wonder if there is an algorithmic way to decide when two crate names are 'near' each other. Then, if you added a crate with `cargo add` and there is another similarly-named crate with much higher usage, a warning could be emitted.
*EDIT* I know there's already https://en.wikipedia.org/wiki/Levenshtein_distance, but I wonder if there is a better measure that looks at e.g. keyboard layouts and likely typos. I'm sure there will have been research done on this.
That way "ripgrep" could include "rg", searching cargo for "rg" brings back "ripgrep", not a second package named "rg", and an install could tell the user the correct name for any attempt to install it.
This also covers typo-squats, so there would be no need for packages like "memap".
Obviously this represents a low-effort vector for massive squatting, so maintainers would need to be responsible for preventing that, and could add some typos themselves, being the ones which see the request for the mis-typed packages.
println!("You meant to install ripgrep: type `cargo uninstall rg` followed by `cargo install ripgrep`"); compile_error!("You meant to …");
Eg:
https://github.com/rust-lang/crates.io/blob/58e505f2abdabd6a...
https://github.com/rust-lang/crates.io/blob/58e505f2abdabd6a...
https://github.com/rust-lang/crates.io/blob/58e505f2abdabd6a...
https://github.com/rust-lang/crates.io/blob/58e505f2abdabd6a...
Not sure how to feel about this... on an individual-package level, it seems a sensible enough idea, but if it becomes a widespread practice, the namespace could get really cluttered.
Namespaces are a solution or mitigation to some problem, but that problem is not malicious typo-squatting.
e.g. bevy and amethyst have claimed a load of crate names like bevy-x or amethyst_y because they thought they might release an official addon to their framework to handle those areas. e.g. bevy did it with this https://github.com/bevyengine/bevy-crate-reservations/blob/m... and amethyst did it the long way as far as I know
Organisations wanting to have consistent package names and users wanting to identify related packages are smaller problems than the "all the good names are taken" and "packages can impersonate other packages with typos" problems.
I will also say this: at the level of personal preference, and given my understanding of many other package ecosystems, I would have preferred namespaces from the start. But I don't feel very strongly one way or the other to be honest.
There is a related RFC open: https://github.com/rust-lang/rfcs/pull/3243
> "packages can impersonate other packages with typos" problems
I was pointing out that this is specifically not solved at all by namespacing. A package's name includes its namespace, and the namespace can be typo-squatted. (EDIT: Or wait, maybe I'm misunderstanding what you're saying. Perhaps I'm confused by what "these" refers to in your last sentence.)
The only problem I see is that I don't know who owns ripgrep, so sticking some random name in front of it only adds to my confusion.
I think I'm just going to refrain from talking about namespaces now. I tried to be super vague and just point out one thing that wasn't solved by namespaces (typo-squatting), but I've somehow gotten sucked into this discussion.
No more. I'm done. Sorry, it isn't directed at you. Just tired of the discussion around namespaces and the neglect to account for the "admin" parts of it.
Crates.io is incredibly cluttered with namesquatting. It’s probably the worst package registry for it, even surpassing NPM.
Part of the problem is that they explicitly say name squatting isn’t against the rules.
This installs a library by some authors not affiliated with AWS.
Instead of: `pip install awscli`
Which is what you expect.
I can't believe that a good way to see what's inside is to make a rust project, add the crate and then go searching around the local filesystem.
I usually use lib.rs instead: https://lib.rs/crates/rg
That has a link to source: https://docs.rs/crate/rg/0.1.0/source/
And here's the Rust code: https://docs.rs/crate/rg/0.1.0/source/src/main.rs
This one just depends on the correct `scikit-learn` package though.
(and `pip install beautifulsoup4` lets you `import bs4`)
> You tried to install “pytorch”. The package named for PyTorch is “torch”
https://github.com/fregante/npm-helpful-typosquatting
Here’s what it looks like: https://www.npmjs.com/package/webext
I love Python but pip/pypi and imports always felt wierd to me because of namespaces, package names, special imports "as", etc., maybe this is a bias because I started using them when I was younger and now I'm more experienced, I already know how to use most package managers.
BTW Ripgrep is awesome, I'm learning Rust and it's an inspiration to me, thanks burntsushi!
I can imagine for example, importing keys from only the authors that I think I can trust, and passing a flag to cargo that only allows using those packages for cargo install or cargo add.
In this case I think just checking the top level crates signature (and not dependencies) would be enough to mitigate a lot of issues including typo squatting.
Let people do the mistakes once and learn the correct package name, instead of relying on a hack and potentially introduce confusion later.
Better to just make `cargo install rg` fail so that it never worked in the first place. `cargo install ripgrep` is also more self-describing and gives you a better search engine query.
The solutions here are non-flat namespacing (which has worse UX, since `cargo install some-tool` now becomes `cargo install whats-their-handle-again/some-tool`) or some kind of content addressing (which is similarly bad for UX, if not worse). Most package indices choose neither, and "solve" the problem by playing whac-a-mole with abuse instead.
This means the first package to squat on the name can use the shorthand version, while allowing other packages with the same name in other namespaces. (which may be forks or entirely different packages)
Edit: Because I'm on a Zoom call that will never end.
"ripgrep is a line-oriented search tool that recursively searches the current directory for a regex pattern. By default, ripgrep will respect gitignore rules and automatically skip hidden files/directories and binary files."
Unfortunately it defaults to parsing a git tree's gitignore file and skipping over files listed in it.
Is that true? How could anybody think that this non-orthogonal monstrosity would make any sense?
I will agree that in a sane project setup you don't need to search through all files including build artifacts ignored by git.
Orthogality is a means to an end, not an end itself.
The idea behind it is that it acts a heuristic for reducing false positives from your search results. For example, ripgrep replaced several little grep wrapper scripts I had in ~/bin.
And fortunately the default behavior is easy to disable. `rg -uuu foo` will search the same stuff as `grep -r foo ./`, but will do it faster.
I recently came up with this alias to make ripgrep do what I want: do not skip hidden files, except for the .git directory:
alias rg="rg --hidden --glob '!.git/'"
(Note: if you try entering this alias interactively, you may have to escape the '!'...)
* The repo can add a `.ignore` or a `.rgignore` whitelisting things like `.github`. ripgrep will pick up that whitelist automatically and search `.github` even though it's hidden. But this relies on the repo adding ripgrep-specific config files, which is maybe not so realistic. (Or not universal enough to rely upon.) But it could work fine for repos under your control.
* Add '!.github/' to, e.g., `~/.config/ripgrep/ignore`, and then add `alias rg="--ignore-file ~/.config/ripgrep/ignore"`. That will become a global ignore file (with low precedent) that will whitelist `.github` everywhere.
And of course, thanks for this wonderful tool!
There is a ripgrep "config file" (not mentioned in my previous comment), but there is only one and you have to set it via RIPGREP_CONFIG_PATH.
The things mentioned above are "ignore files," which are a sort of configuration for whitelisting and blacklisting files to search in a directory tree. And yes, you can splat them down into any directory, and if ripgrep enters that directory, it will read it and respect it. (Unless you tell it not to.)
If there are a lot of files with a lot of patterns, indeed, that can wind up taking a chunk of time not only building the matchers for each config file, but for actually matching them against every path. Sometimes it takes longer than not ignoring files at all! But if your ignore files are permitting ripgrep to skip GBs of data that GNU grep wouldn't otherwise skip, well, that's going to be a huge win no matter how you slice it.
ripgrep does use multi-threading as well, and it makes sure every ignore file is parsed and built only once. All other threads can then share the one single matcher.
That's a feature.
Like, it's the entire point of ripgrep. It's designed to search through the things a developer actually cares about searching through.
If you actually want to search everything, just use grep.
If you have `ripgrep-team/ripgrep` rather than `ripgrep`, it doesn't help at all with people typing the wrong thing, like `rg-team/rg`. I fail to see how it helps.
It's even worse with packages that are (currently) authored by a single person, how many people know the name of ripgrep's author? Or rand? Or bevy?
If someone can find a meaningful case where ag is faster than ripgrep, then I'm happy to accept a bug report. I'll do my best at that point to give an analysis of the benchmark, and if it's correct, I'll either try to fix it or say why it's hard to fix.
By "meaningful" I mean "something that is noticeable to humans." So for example, reporting a bug because ripgrep took 9ms and ag took 7ms on a tiny repo is one I would consider not meaningful. :)
(Sorry about the verbose caveats, but just trying to head off responses I've got in the past.)
Moving to a different set of quirks is not a step forward than continuing to use the ones you know. Regexes aren't de-facto standard; grep is different to egrep to perl to python to C++ to your text editor to whatever. It's a massive pain and annoying as fudge. You're a programmer you know this. A "superior" set of quirks may be better for new regex users but it is worse for everyone else who now has to know both grep (and all the other regex quirks) and ripgrep if they're going to use it. To get this done like I always have I now need to know something new. A new user doesn't care about the obsolete.
Faster? Well I have not yet experienced an issue with the speed of grep, that's my experience. I can imagine this could be compelling for uses I don't know about.
ripgrep may well be a better grep for some users. And that is Great, really! We should all try and make things better! Hurrah!
Refusing to describe how it is different and why you might like to install something non-standard (for which there could be compelling reasons) is just silly. Hyping anything at all in that context like that looks pretty bad.
The ratio of content-free hype (omg ripgrep is fantistic!) to an actual description on this thread or in the link or seemingly anywhere I clicked is pretty bad and constitutes a signal.
More to the point, the comment you were responding to said "yeah but with less quirks." It didn't say "no" quirks. So your comment ended up being a silly non-sequitur.
Your follow-up comment looks ever worse to be honest, and sounds like an argument for never building anything different at all.
And half your comment is whinging about hype. Really? Yeah people get excited about shit that helps them get stuff done more pleasantly and faster than before. Who would have thunk it. Some great mystery.
Your comments get a big thumbs down from me (author of ripgrep).
Readme was not linked here. I looked and didn't find it in the parent link. Feel free to abuse my competence or, you know, link it in the discussion? Actually I'll do it.
https://github.com/BurntSushi/ripgrep/blob/master/README.md
There. That feels better doesn't it?
More quirks you have to learn is not less.
You know 5 quirks.
This new thing has 3 /different/ quirks to perform the same task you already know.
To use the thing with "less" quirks you need to know _more_ quirks. And that can be fine! Or you might decide it as not worthwhile. But it is more quirks you gotta learn. Pointing that out is not a non-sequitur to a discussion that starts with:
>So it's grep like i already have installed and know its quirks?
"Yes, and it's faster if you need that and its ui makes more sense to me." Is an appropriate addendum.
Less quirks means you gotta learn more. And that is just life for all of us and one of the barriers that protects incumbents that are in some sense not as good. Note my selfish comment was taking my perspective that I already have grep and I already use it and know its quirks.
From your readme:
>You need a portable and ubiquitous tool. While ripgrep works on Windows, macOS and Linux, it is not ubiquitous and it does not conform to any standard such as POSIX. The best tool for this job is good old grep.
So you see how if you have to know grep anyway, less quirks isn't a great thing in and of itself to the user. Simpler to use interface, yeah that could be compelling. Or not. Faster could be compelling. Or not. Some additional feature could be compelling.
Maybe ripgrep is amazing and the future and just dandy and I wish you the best of luck with everything you want to achieve with your version of grep.
--------
>Your follow-up comment looks ever worse to be honest, and sounds like an argument for never building anything different at all
That's basically pretty rude and difficult to reconcile with what I actually said.
>ripgrep may well be a better grep for some users. And that is Great, really! We should all try and make things better! Hurrah!
Q. What is ripgrep?
A. A faster grep where some people find the defaults in the interface more usable, with some potentially interesting additional features.
What a lot of words wasted to get to that Q & A. Please feel free to improve it while maintaining that length. Signal to noise.
>>> So it's grep like i already have installed and know its quirks?
>> yeah but with less quirks and runs much faster. worth trying out
> Uses regexes = has quirks.
Then you respond with "uses regex = has quirks," as if the parent said "ripgrep has no quirks." Well, no shit sherlock, any grep is going to have regex support and any grep is going to have the quirks that come with regexes. That doesn't mean what the parent poster said was incorrect and it certainly doesn't mean the parent poster implied ripgrep had zero quirks. Your comment was a refutation of a silly straw man.
Real subtle, I know.
> Readme was not linked here. I looked and didn't find it in the parent link. Feel free to abuse my competence or, you know, link it in the discussion? Actually I'll do it.
If this was a link to a blog post about ripgrep or a release announcement or something of that sort, yeah, absolutely, the author should have given ripgrep at least a short introduction and a link to the README. But this is just a link pointing out a typo-squatted crate meant to improve failure modes. It's a "hey here's something interesting." It isn't even interesting just because of ripgrep, it's interesting because some person cared enough to try and improve failure modes and prevent someone from installing the wrong thing. If you want to know what ripgrep is from there, it's on you to go figure it out. It isn't on your fellow HN commenters. They explained what it was in their own words and you didn't like it. And now you've spent who knows how much time here whinging about "herp derp hype and but but but signal-to-noise" when it would have taken you less than 5 seconds to Google the damn word and click on the first result.
> A faster grep where some people find the defaults in the interface more usable, with some potentially interesting additional features.
LOL. This is a terrible description. ripgrep's official description is shorter and far more useful, because it gives you an idea of what it's actually doing, instead of just vaguely talking about it:
> ripgrep recursively searches directories for a regex pattern while respecting your gitignore
Bottom line is:
* You made silly snobbish comments.
* The rest of your whinging could have been mitigated by a 5 second Google search. Yes, normally, for things like release announcements[1], I include a short blurb about what ripgrep is. But this wasn't a release announcement. You want the official answer? Then go look for it. You want the (possibly described in terms of their own experience) opinion of your HN peers? Well, that's what you got. Live with it.
[1]: https://github.com/BurntSushi/ripgrep/releases/tag/13.0.0
so (i) grep with the -r flag by default and (ii) has the feature of reading your .gitignore when present which is potentially interesting. But yours is still a better answer than "searches for stuff" and the many other answers here where nodobdy was saying /why/ they liked it. The tenor of that conversation on this site was the objection and remains so.
You seem to be taking this very personally and indulging in a bit of flaming of a kind @dang doesn't care for. I don't mind though. Yet again I say good luck to you.
There's even a huge sign with only 12 words pithily explaining what the shop has inside.