Copying stdin to stdout in Java | Hacker News Reader

Copying stdin to stdout in Java

gist.github.com

58 pointsKostaC57 comments

As he notes in the comments, you'd really just do this:

IOUtils.copy(System.in, System.out);

using the Apache Commons library.

That's the most apples-to-apples comparison to more "scripty" languages that have more stuff built in. With Java you pick the libraries/frameworks you want to use. Commons and Guava are extremely popular and robust libraries and really should be included in any comparison against "real world" Java.

I actually like how in Java you can get down closer to the metal and control the exact buffer size if you want to, or handle all the exception cases instead of just passing them up if you want to. If you're comparing intentionally lower-level more verbose Java code to higher-level Python or whatever, that's not really apples-to-apples. High-level Java involves frameworks.

Still, Awk and Perl need no library, the semantics was the part of the language since the beginning, and is always available to every programmer, also for the bigger programs. Awk and Sed were obvious inspirations for Perl.

Awk and Perl are tools built on another language that does stream copying i.e. read stdin and write stdout.

You could build Awk and Perl in Java if you wanted and have the same result.

  System.out.println("hello world");

Is the same as:

  printf("hello world\n");

>>High-level Java

How is file manipulation High level. That sort of stuff should work right next after your Hello, World program.

If you are telling me Java intentionally makes easy things difficult. Its the last thing I want to use to solve difficult problems.

File manipulation, like any kind of computer programming, can be programmed in a high-level or low-level way. High-level is easier and reflects some assumptions and opinions. Low-level gives you more control. With Java, you can choose which way you want to go, and the easier route is only as "difficult" as choosing a high-level framework.

That's right, but it seems like it's fashionable to bash Java inventing example code that noone uses in prodcution. If I would see the code like this, I would ask the one that has written it to justify why he didn't just use IOUtils.copyStream method. Is there some performance reason he wanted to specify buffer size manually? Maybe he wanted to handle exceptions properly instead of wiling silently? Or he was on extremely constrained device where he couldn't afford to include 40k library and just copied the only method he needs instead?

It's not just Java that is bashed with this tactic. For example, the TechEmpower benchmarks that keep getting reposted here do the same thing, except with languages other than Java, in order to make Java come out looking better.

That's getting a little off topic, but the TechEmpower benchmarks are open source and open to submissions. If you see a problem and you haven't submitted a pull request then you have no excuse for complaining.

If you want a single point generalization about Java programmers, I was one for years, and the way I would have implemented this involves minimally a BufferedReader and an InputStreamReader. You'll also probably need to catch the IOException. Plus or minus 20 lines or so.

I don't hate Java, but "Java as it is spoken" does not often get used for Unix-style piping in a chain of small programs on a command line. It also isn't the easiest tool in the drawer if you want to do light (or heavy) string processing.

Sure, but I wouldn't bring my forklift out of storage to move a sack of potatoes across the room either which is what you're doing when you use Java for a 10 line shell script. That's not what Java is designed for.

Nor you grab an camel when you want a ride, as there are better options.

Wrapping System.in in a Reader is exactly the wrong thing to do. You want to copy the bytes directly, not guess what the character encoding might. The input might not be characters at all.

The essence of the argument here is that Java is a poor language as it doesn't offer simple abstraction to copy standard input into standard output; that Java is too verbose and low-level for the task.

The lack of direct abstraction is not a valid argument, because as a programmer, you shouldn't be writing the logic in Java, Python, C or Scala but rather a higher order domain language implemented in the chosen host language and that's what the process of programming is all about. For the majority of real world programming tasks it's unlikely that a language that fits the domain perfectly already exist, so you have to create one.

In Java one can say:

    copyStream(System.in,System.out);

And then one will have to implement copyStream but only once:

    long copyStream (InputStream src,OutputStream dst) throws IOException {             
        long bytesCopied;
        byte[] buffer = new byte[8192];
        int bytesRead = src.read(buffer);        

        while(bytesRead!=-1) {
            bytesCopied+=bytesRead;
            dst.write(buffer, 0, bytesRead);
            bytesRead = src.read(buffer);
        }

        return bytesCopied;
    }

I prefer programmers taking this approach of implementing domain specific language first and then expressing the logic in its terms instead of trying to express higher order concepts without resorting to available host language abstractions.

Let's say Python or Perl let one express stream copying more concisely straight out of the box. However when faced with real life programming challenges one will very quickly encounter limits of what a language can express out of the box with one-liner. But as a programmer one has the power to create one-liners from scratch!

Disclaimer: I am not a Java expert, so the code above is just to illustrate the idea based on my very limited knowledge of Java.

Two things.

First, constantly having to write implementations like this introduces a ton of friction as compared with reusing existing implementations. Languages do have an influence on that.

Second, languages vary in "expressiveness," determining how much code you have to write in order to make an implementation like this. In one language, copying stdin to stdout fits into a natural idiom which also handles other cases naturally. In another language with less care for ergonomics, every task might be equally un-idiomatic.

In other words, a language can offer its own "higher order domain languages" for the core tasks that everyone is doing over and over again as part of their general programming. Or it can choose not to do that, because what it offers is already Turing complete. But then the ergonomics are bad, and it makes a real difference.

It seems wrong that when I am paying for things like long JIT warmups and stop-the-world GC, I am still writing piles of functions with low-level idioms that are no more expressive than C's.

If Java ships with a 'copy from one stream to another' primitive then I'd find that a much more compelling argument than that I can treat myself to reimplementing things like stream copying, sorting and basic data structures on a regular basis. It's unbelievably tedious and wasteful to do this, there's just no reason.

Agree, in Java (and any other language) the main culprit is not the complexity that can eventually be abstracted away, but the trivial noise that cannot be abstracted at all: verbose class definitions, clumsy anonymous functions prior to SE8 (functors), lack of implicit interfaces, lack of infix method call notation, generic type erasure, lack of basic type inference, lack of continuations etc.

My point was that limited expression means of a language are sometimes compounded by inability of a programmer to make a good use of the expression means already available to them.

I also understand the desire for more a expressive language, I am a programmer after all. One has to keep in mind, however, that the more expressive a language is the harder it is on the reader. Java code is trivial for a reader to follow (if not for the excessive verbosity sometimes covering up the true intent); much of Scala code base, on the other hand, is not that trivial to comprehend due to the high expressiveness of the language.

I don't know much Java. Does 'System.in.read(buffer)' block until stdin is at EOF or buffer is full? Then this solution would be somewhat broken. Imagine a low-bandwidth source on stdin and a audio player connected to stdout, like

  wget -O - http://audiostream | java inout.java | mplayer /dev/stdin

If I had to write this in C I'd use file descriptors in non-blocking mode and select(). With Tcl one could use event-based IO.

Using non-blocking mode may be better, but to answer your question: yes, it does block [1].

[1] http://docs.oracle.com/javase/7/docs/api/java/io/InputStream...

(You referenced the one byte read function, instead of the multi-byte function used in the OP.)

The documentation does not answer how long it blocks. Blocking when there is no input is completely ok in this use case as there is nothing better to do instead of waiting for input. The relevant question is how long it blocks after there is some input available for efficiency reasons.

> there is nothing better to do instead of waiting for input.

In case no more input is available it would be best to already output the data that have accumulated in buffer. Else in case the input program deadlocks you're never going to see the last (up to 8191) bytes output.

This is what non-blocking I/O is for. With non-blocking I/O you first wait until your input channel becomes readable, then reading functions return as soon as the data available for read is exhausted.

You are right but I was talking about the case when no input was there to be read since the call to `read` and also about this specific application:

> Blocking when there is no input is completely ok in this use case as there is nothing better to do instead of waiting for input.

(Emphasis is new.)

You're right, I overlooked the reference to the function reading a single byte.

The documentation [1] does not mention any buffering unfortunately. In a quick test on the console it showed line-buffering behavior, so it probably depends on the input file type?

[1] http://docs.oracle.com/javase/7/docs/api/java/io/InputStream...

Not knowing the exact details of used classes, the bigger question is if the exceptions in the OP code can be thrown and if some try catch is needed. In Java the classes like to throw a lot.

Python, shell, etc. isn't suitable for high scalability, fast applications (e.g. Apache Solr, Lucene, Hadoop etc.). And beginner examples like these make no sense. Java clearly has an important role in this world, maybe not in yours (if you're so irked by "verbosity").

Are you absolutely certain that nobody has used Python in high scalability, fast applications?

I don't have the post context, but at 13 lines, I feel even something like Go wouldn't even be considerably larger or marginally easier to understand.

It's not much shorter, but it was pretty easy to write it you know the libraries.

    package main

    import ("os"; "io")

    func main() {
        io.Copy(os.Stdout, os.Stdin)
    }

The order of os.Stdout, os.Stdin is interesting. I'd assume src->dest being more intuitive order.

Any ideas if this dest<-src order is by design following go design principles or was mainly authors' personal choice?

By the way, Pipe in the same package is the other way around: (PipeReader, PipeWriter)

Channels in go use the exact syntax you mention (dst <- src). With this in mind, the argument order of io.Copy is consistent, as well as io.Pipe. You read from a reader - the output of the pipe, and write to a writer - the input of the pipe.

memcpy() and strcpy() and a whole bunch of functions inspired by them have take (dest, source). Maybe it's not theoretically superior, but it is familiar to a whole lot of programmers.

It is by design. All those work in the same direction:

    dest = src // Variable assignment.
    <-ch; // Read from channel.
    // in net/http
    type HandlerFunc func(w ResponseWriter, req *Request)
    // in io
    func Copy(dst Writer, src Reader)
    // builtin
    copy(dst, src) // Slices.

So the reason is consistency. I suppose the Pipe exception is to map Unix's pipes.

Amazingly, in Java 8, they added a Files utility class to copy an InputStream to a Path, and copy a Path to an OutputStream, but no Files.copy(InputStream, OutputStream).

At this point, Guava should just be added as part of the JDK. :) In Guava, ByteStreams.copy(System.in, System.out)

Java has really solid dependency management with maven, so I'd actually like to see less in the JDK. Let Guava/JodaTime/log4j/etc. keep their faster release cycle, just deprecate the parts of the JDK that used to do those things and point developers in the right direction.

There are some comments on the Gist claiming Python or Ruby is a one liner. My response: https://gist.github.com/kosta/9777932/#comment-1199293

Your Python code is just wrong. I'm not sure what you think it's proving when you write explicitly wrong Python code, and then show that the output is wrong.

>Your Python code is just wrong.

Could you elaborate? I want to understand how to fix it.

EDIT: I really want to know what is wrong with my code. It works. Is it a stylistic concern?

    $ dd bs=1m count=10 if=/dev/random of=randomdata
    10+0 records in
    10+0 records out
    10485760 bytes transferred in 0.878533 secs (11935535 bytes/sec)
    $ python stdin_to_stdout.py < randomdata | cmp - randomdata
    $ echo $?
    0

EDIT2: I think I know how you came to the conclusion my code is wrong. Perhaps next time you should read the __entire__ post. Thank you.

more C-style

  while( (bytesRead=System.in.read(buffer)) != -1 )

may want to link to the particular post in the gist :)

Yeah – what post?

"Class that copies stdin to stdout, as compained about as not being cleanly writable in Java on Hacker News."

That post, maybe?

"Why I like Java" by Mark Dominus:

https://news.ycombinator.com/item?id=7463671

Who never wrote it but assumed we know that the same program in Perl, without using command line switches is

    while (<>) {
        print;
    }

Still Awk is shorter:

     { print }

And if you are allowed change the switches of the command line to Perl for your program that Perl program can have exactly 0 bytes.

Awk is even shorter: anything truthy works. You don't even need the print statement or brackets. Thus there are nine separate 1-byte programs which produce cat: [1], [2], [3], etc etc.

By default, unless you use the BEGIN block or something, awk will run your program on each line of stdlin. This is useful for programs of the type:

(/some regular expression/) { some action}

The default action if you don't specify one is "print $0" (the whole matching line). If your condition is a plain-ol' expression rather than a regular expression, and it always evaluates truthy, you thus get every line.

You are right, the shortest awk is one byte. Please post the link all nine variants I wasn't able to Google it.

Still the shortest Perl is 0 bytes when the command line switch is allowed to be -p

Update: That's actually called "the sed mode of Perl." Moreover these two command lines behave similarly:

    perl -pe 's/search/replace/g'

and

    sed 's/search/replace/'

SED is shorter (0 byte). SED /is/ a turing complete language [1] so don't try to claim that I'm cheating.

  sed < input > output

[1] http://robertkotcher.com/sed.html

[update] s/set/sed/

The command line equivalent, with Perl:

    perl -pe '' <input >output

Even java can do that setOut(system.in); (iirc)

> Even java can do that setOut(system.in);

I was referring to an empty SED-program that would then be run via 'sed -f empty-file.sed'. Of course you need >0 bytes to invoke the SED program.

Sure, but if you want to modify this code to send stdin over the network or through bluetooth in the Java version, you would just replace 'System.out' with the socket. The code would remain the same - in fact the whole class could be reused by having it pass in an input and output stream as parameters.

Your example is very much just a special shortcut for one special condition. It's not generalizable in the same way. And let's be honest - most of today's code is not writing to standard output, it is writing to some iOS or Android GUI or outputting JSON to be used in some javascript webapp. That kind of shortcut just doesn't make sense anymore and is a relic of a different time.

In Linux you can redirect the stdout to the the socket outside of your program, no programming required. You can test and develop everything with stdout and let the same unmodified file run in the web server. See traditional CGI, that's how dynamically generated web pages were easiest to implement long ago:

http://computer.howstuffworks.com/cgi3.htm

The server opens the socket, redirects the stdout to his socket while calling your program fully unmodified.

You open network sockets in Linux and redirect stdout to them? It's possible with curl I guess, but generally people don't do that as it's hard to send the data in the right format. Mostly people will use a tool to access network services.

As for serving files, you'd normally let nginx or apache take care of the sockets for you and not use output redirection.

In Plan9 it's actually viable to use network sockets as files which is pretty nice, but for Linux it's not really how people do it.

> but for Linux it's not really how people do it.

Please explain that to all the people who wrote CGI scripts for years. Seen the link at all?

CGI scripts are dead though. It turns out it wasn't a good way to do it. It's better to use a nice new invention like Apache or Nginx to handle the sockets for you and then use frameworks specifically designed to handle HTTP responses as piping HTML through std inputs is difficult and error prone.

The ability of a language to easily write to std out is simply not important anymore apart from shell scripting which is mostly used these days for setting up the runtime environment and building other languages to handle the grunt work.

I don't know of any companies that still pipe std output directly to sockets anymore, but I guess there must be some. If your company still does this then a language feature like that makes sense. For most people, having the ability to write code that can directly write to sockets without piping through a shell is better practice.

I wouldn't say they are totally dead. They are great for relatively quick/dirty code.

The major advantage that CGI gives you is the fact that you get a new, clean process on each request. This means you have fewer and simpler failure cases. This can be helpful when you have a line of business web application though it is certainly suboptimal for public-facing scalable systems.

I wouldn't say cgi is dead. It still has a role to play. It is not the ideal way for doing many things though. I.e. they are no longer the go-to tool, but rather one tool for relatively lower-volume applications where performance matters less than some other considerations.

grep is even shorter:

    grep $

That reads lines from stdin and echoes to stdout.

Grep expressions are not a turing-complete language, so I guess that won't count. You could claim that you solved the task using a shell script (that invoked the grep "function") but if you programmed in shell, the shortest solution AFAICS is

  #!/bin/sh
  cat

That mangles binary data.

    $ printf 'binarydata'|awk '{ print }'|hexdump -C
    00000000  62 69 6e 61 72 79 64 61  74 61 0a                 |binarydata.|
    0000000b

Reply on news.ycombinator.com