Processes Paralleling to Speed up Computing and Tasks Execution in Linux | Hacker News Reader

Processes Paralleling to Speed up Computing and Tasks Execution in Linux

kukuruku.co

47 pointsandreygrehov36 comments

> Besides, xargs isn’t good at transferring special symbols, such as space or quotes.

Well, no, not when you're piping ls into it. That's what --null (or -0) is for. (Use find instead.) The man page offers a similarly baffling argument:

> xargs deals badly with special characters (such as space, ' and "). To see the problem try this:

  touch important_file
  touch 'not important_file'
  ls not* | xargs rm
  mkdir -p "My brother's 12\" records"
  ls | xargs rmdir

Same problem as the article: piping in the output of ls into xargs.

> You can specify -0 or -d "\n", but many input generators are not optimized for using NUL as separator but are optimized for newline as separator.

I still don't see the argument: if the generator is "optimized for newline as a separator", then -d is what you need. But if your generator is outputting a list of file names (perhaps the most common case of xargs), then -0 is what you need. It is perhaps somewhat unfortunate that its use of whitespace (and bash's use) make it so easy to write subtly incorrect programs; this is perhaps why there's such common advice to use Python, or anything not Bash. But the argument above leads me to think that `ls | parallels` would somehow work, which it cannot, for the very reasons we must use xargs -0.

I suspect this is a long way about saying "we have (somewhat more, but still not quite) sane defaults".

The correct command has always been `find . not* -print0 | xargs -0 ~/code/random/args.py`. Anything else is subject to weird corner cases of failure. It's a sad state of affairs really, and I'd love to change it. If you're just pounding out a command on the command line, you can usually ignore those corner cases, as you probably mentally know whether `ls not*` is going to cause issues or not; if you're writing a script for general use, then it's only a matter of time until it fails in interesting ways.

> The correct command has always been `find . not* -print0 | xargs -0 ~/code/random/args.py`

If you need to be portable, you can do pretty well with

  find ... | sed 's/./\\&/g' | xargs command

This will only break with filenames that have newlines in them.

(see http://www.etalabs.net/sh_tricks.html)

Use pipelines with simple filter commands. No thread programming required. Use ssh to distribute this across nodes securely, assuming the I/O doesn't make that unworkable.

The 'make' trick is cool. Used that a couple of times. I just kind of discovered that on my own. Then someone told me about xargs, so now that is my default.

Sometimes by playing with the -j or xargs --max-procs you discover interesting bottlenecks. Remember spending time trying to speed up this script. Used the make trick, it still was not running fast enough, it turns out disk i/o was the bottleneck.

I don't know enough about make to explain, but sometimes -j will break the build. Some Debian packages are explicitly built without -j enabled.

Sometimes there are incorrectly specified dependencies between source files and things that depend on them. For example rules that are implicitly sequential in -j1 mode but if they are written as

target: dep1 dep2 then dep1 and dep2 might be executed in parallel if -j>1

There are not many make expert (I am not one) so looking and figuring out the issue is sometimes tricky so often I see -j1 used always by default.

I also like the simple option to spawn processes in a loop in bash and then wait.

http://stackoverflow.com/questions/356100/how-to-wait-in-bas...

Here's a bash script for running scripts in parallel (which I use in production at work).

https://github.com/jchavannes/procman

Yes, running in background and checking the status of the pid is coming to mind when one has to do that Bash. I always feel that it's stretching beyond the normal level of usage. For example what can you do with the output? There is no way to sync it unless you'll capture it in a file and analyze later. In cases when I need to write something parallelized I feel that using Bash isn't the best idea.

Yeah, I agree, but there are some use cases for a simple bash script. I updated the GIT repo with a PHP version that has output and errors like you said.

why doesn't the person just use 'time' to show how long it took instead of saying off the cuff 'saved X seconds'

There have been 14 posts from this site (not counting deleted ones) and most appear to be unauthorized, uncredited translations from a Russian site, habrahabr.ru. Some of the Russian texts were themselves translated from English. See the discussion at https://news.ycombinator.com/item?id=7754641.

Should we ban kukuruku.co? There's good content (if not good English) in the articles, but these tactics are sleazy and it doesn't feel like behavior HN should be involved in encouraging.

https://news.ycombinator.com/item?id=7770909

https://news.ycombinator.com/item?id=7770887

https://news.ycombinator.com/item?id=7770727

https://news.ycombinator.com/item?id=7768391

https://news.ycombinator.com/item?id=7765671

https://news.ycombinator.com/item?id=7764938

https://news.ycombinator.com/item?id=7763187

https://news.ycombinator.com/item?id=7753792

https://news.ycombinator.com/item?id=7753166

https://news.ycombinator.com/item?id=7741937

https://news.ycombinator.com/item?id=7734202

https://news.ycombinator.com/item?id=7732049

https://news.ycombinator.com/item?id=7727016

https://news.ycombinator.com/item?id=7702566

I'm torn. I submitted a couple of those articles, and was embarrassed when the original author showed up in the comments to mention that his article was being used without his knowledge. On the other hand, some of them are really good articles to which I otherwise would not have access due to my limited knowledge of Russian.

It's awkward, but in the end, I lean toward keeping them here, promoting their translations, and encouraging them to give clear credit to the original authors. While banning them might feel right, I fail to see it making the world a better place. I think it's better in the end to keep them here and gradually encourage them to improve their ways.

Please ban them, this is absolutely unacceptable. The original author put hard work into his writing, and I still don't see any credit in a majority of their translations. They have published 14 articles in total without any credits going to the original author. It's definitely not the type of news I would expect to be associated with a "hacker". I do not see any links to the original source at http://kukuruku.co/hub/cpp/lock-free-data-structures-introdu...

They are stealing content.

Another user (leephillips) posted the following about their posts: "I went to the site and left a comment complaining about how they stole your article. They deleted the comment, and you still aren't credited. Does HN have a blacklist of sites that steal content or publish only blogspam? I think they should consider it." - https://news.ycombinator.com/item?id=7754641

I understand how you feel. But to be fair, the author of that translation has said (on HN, as well as in a reply to an email I sent him) that he didn't mind it being used and that it was itself a translation of an English article to begin with.

The comment was deleted due to filthy language in a comment. We'll add links as soon as possible. No panic, please.

The issue with articles has been fixed. We specify original source! We only share useful content on HN, not all of the articles. In addition, we don't break Habrahabr's agreement. There was one disappointed user, who mentioned that the "original source" was not specified. We fixed that and since then set the source (if we use it) everywhere.

One worry is that the copyright of the piece remains with the original author (http://habrahabr.ru/info/agreement/) and that publishing a translation without explicit permission from that author violates international law. Providing a link to the original helps to show that your intent is good, but unfortunately does not change the legal situation.

I do agree that you are providing a useful service by spreading high quality technical information. My understanding is that unauthorized translations are more common and more tolerated in Russia than in the US. But HN is a very law-abiding business-oriented site, and justifiably is concerned with following the law. Thanks for signing up here to explain, and please try to work with DanG to come up with a plan.

Here is what agreement says:

"Пользователь осознаёт и соглашается с тем, что размещённая им информация может быть использована поисковыми или другими автоматизированными сервисами и иными способами."

Here is the translation:

"User acknowledges and agrees that the uploading of such information can be used by search engines or other automated services, and in other ways."

But anyway, we will get in touch with authors. We'll try to find that out.

DanG: let me know if you guys have any other concerns. Do you want us to specify the link to author's habrahabr profile in addition ?

I still don't see credits going to the original author and / or website for most of your article beyond a very easy to miss [original source] at the bottom. Many articles have no references to the original author at all. It should be very clear at the top of the article that your work is a translation from the original author. The original author's website should have a very prominent link, at the top. It really surprises me that you are knit-picking at site agreements, where is the hacker integrity?

I think that when you're publishing translations then, at a minimum, you should make clear at the top of the page that it is a translation, provide a link to the original, and include the author's name or username. You should also make a good faith effort to seek permission from the author.

If the article you're translating was itself translated from an original source, then "author" here means author of the original material. The latter would seem like an obscure edge case, except that some of the translations you published actually were that.

We want solid content on HN, so I'd like to find a way to have these posts not be a problem. No doubt the community will scrutinize future posts, so we may have this discussion again, but for now we won't ban the site if you do the above.

Original articles, of course, are more than fine. One problem with grey-area tactics is that they make your whole site look shady. Readers will think that all your articles must be hijacks, whether they are or not. To pick one example that was posted to HN, who wrote http://kukuruku.co/hub/mobiledev/mobile-typography? It's not clear whether this content was created for your site or stolen from elsewhere; besides which, readers want to know the names of the authors they're reading.

If you attribute everything clearly and prominently, this should go a long way toward clearing things up.

Hasty Edit: Googling reveals that you copied the mobile typography content from http://my.safaribooksonline.com/book/programming/mobile/9780.... That's disgusting, and certainly explains why you omitted any attribution. If you keep doing it, we'll ban your site, despite our fondness for translations of obscure but solid Russian technical articles.

It's in your interest to pay attention here. You have first-rate, highly technical HN users like nkurz, acqq, and jacquesm—plus a site moderator—bending over backwards to help you. If the material on your site were merely the usual blogspam, none of us would care. We're putting time into helping you because we want access to the solid material. There's clearly demand for top-notch Russian articles in English translation. On the other hand, none of us has patience for yet another shitty hijack outfit on the internet. You can't be both; not here, anyway. Please turn this into a positive by listening to the community feedback and cleaning up your act.

Ok, thank you, Dang. We'll take care.

Last I checked, you specify "original source" with an exceedingly obscure link at the bottom. Why don't you credit the authors and the original source properly at the top?

Also, what about asking authors for permission to translate and republish their work?

Well, first of all habrahabr does not share real names of their users. So specifying something like "TR156" does not make any sense. As for the link, we checked a lot of web sites and it seems most of them specify the source at the bottom of an article. The link we add is not obscured. It has exactly the same font size as the rest of the article's content. We also wrap it with square brackets to make it more noticeable. As for permission, there are two things. 1). We don't break Habrahabr's agreement, so we don't actually have to get permission. 2). There is no way to get in touch with authors - we don't know their emails. We would be happy to. No one can also proof that the HN user who had complaints is a real author. As long as we haven't received any information from official Habrahabr representatives, we are fine. Right now we also working on our own content a lot. We don't want to publish translated content. But we thought that sharing useful and interesting information from other countries would be helpful for community. In anyway, we are open to communication.

> specifying something like "TR156" does not make any sense

No, your claims are dangerous and misleading. Even if I decide not to show my real name I want to be present as a person who wrote the article. You take that away. The name of the author and the link to his profile before the start of the article are the minimum you have to do. You have to ask the author for permission before you do the translation and publish the article, using the communication mechanisms of the site from which you take the content. And you must write that it's a translated text.

Before you do all that, the way you present your material remain questionable enough to have my vote for banning the articles.

Habrahabr is a private community. There is no sign up => there is no way to get in touch with authors unless do manual research! We are HAPPY to get in touch with them. What exactly sounds dangerous ? We did research, all the popular websites put a link to the source at the bottom of an article. It seems like a normal practice. But correct me if I'm wrong.

So if you aren't a "friend" of somebody on the Facebook that gives you the right to republish the content of his articles on your own web site?

No, it doesn't.

That's different. It doesn't really matter anymore, as we have already implemented "external authoring", but still, if you look on how Habrahabr publishes translations - you will see that they put a small 9px link to an author. 9px ??! it's super small. In addition, I found a lot of translations of those articles into different languages around the web and it seems no one cares. But once again, it doesn't matter anymore, as most of the work is done and we'll add all the links today. Just sayin'.

I would also like to add that the list of urls you posted contains articles that belong to Kukuruku and are not translations.

You are definitely on the wrong side of the derived works line here, and if your articles are not easily identified as yours that is because in my mind that only means the source has been obfuscated enough that I can't easily trace them like you do with the other stuff found on your site, but you're going to have a hard time convincing me they are original now. If you do not have an agreement with the original source of these articles then you are in breach of copyright law. That may be a conscious decision on your part though.

I would definitely support a ban in this case but at the same time I think there is a need for properly accredited translations (good ones, not crappy ones like these where according to one of the authors lots of nuance got lost) of good content found on non English language websites.

Maybe this is something the people behind Hacker News Magazine could look into, they seem to have the right attitude when it comes to re-using content.

We are not going to convince anyone. From now on we specify the source if the content is taken from other resource. That was our fault and I agree with that.

Please, don't spread rumors. What crappy translations are you specifically talking about ?

See comment by real author of copied content displayed on your website in earlier thread.

Suggestion: do not just take content, ask for permission and as long as you don't have that permission remove the content that you took without.

Ok, we'll try to get the permission from real authors by manual research on how to get in touch with them. Fair enough.

It's not simply about getting permission, it is also about prominently attributing credit to the original author.

right, we'll be adding their name.

Reply on news.ycombinator.com