Go can only read 1GiB per Read call | Hacker News Reader

Go can only read 1GiB per Read call

kgrz.io

58 pointssigna1129 comments

Interesting factoid, but I can't see how this would lead to any real world performance impacts.

I am also baffled as to why you would want 2gb reads for wc. System calls are not that expensive and at some point touching so many pages has gotta hurt.

Author here. I wasn't suggesting it would have any perf impact. Just that it was an interesting change set.

I feel I'm missing something here, but it feels like the intro is not only suggesting there was a performance impact, but very explicitly stating there was:

> profile of a sample word count program I was writing, which showed the program was spending way too much time in the syscall module. That in this context can only mean one thing: way too many read syscalls were getting called.

I find it hard to believe that the profile would look any different with 1 vs 2 syscalls per 2GB chunk. The syscall overhead is going to be insignificant compared to actually copying the data. The program is going to be spending a lot of time doing syscalls no matter how many there are, because the individual syscalls will just start to take more time as you increase the size of the buffer.

Edit:

Compare:

  strace -e read -T perl -MFcntl -e 'sysopen FD, "foo", Fcntl::O_RDONLY; while (sysread FD, $buf, 1*1024*1024*1024) {} '

  read(3, ..., 1073741824) = 1073741824 <0.455901>
  read(3, ..., 1073741824) = 1073741824 <0.219711>
  read(3, ..., 1073741824) = 1073741824 <0.213923>
  read(3, ..., 1073741824) = 1073741824 <0.211783>

Vs.

  strace -e read -T perl -MFcntl -e 'sysopen FD, "foo", Fcntl::O_RDONLY; while (sysread FD, $buf, 2*1024*1024*1024) {} '

  read(3, ..., 2147483648) = 2147479552 <0.921789>
  read(3, ..., 2147483648) = 2147479552 <0.487007>
  read(3, ..., 2147483648) = 8192 <0.000031>

I tried your strace commands on my linux desktop and I don't get anything like your results. I get a few lines of 'read(3, "\177ELF\2\1\1...' type stuff. Are you using bash (as I am) or some other shell? Could there be some escaping that I'm missing? Thanks.

It's reading from a file named foo, which in this case was 4GB large and created with:

  dd if=/dev/zero of=foo bs=1024 count=$((1024*1024*4))

You probably don't have a file of that name in the current working directory?

I missed that, thanks.

Makes sense, I guess I misunderstood the premise of the article. It's a good dive.

FYI, factoid has a different meaning: https://en.wikipedia.org/wiki/Factoid

Äckschully no: A factoid is either an invented or assumed statement presented as a fact, or a true but brief or trivial item of news or information. which is exacktly what this is. Innit.

> Since the term's invention in 1973, it has become used to describe a brief or trivial item of news or information.

In the very first paragraph, in fact.

In the past when I have tested buffer sizes with reads I can't say I have noticed much performance difference past really quite small buffers like 16k. I am sure there is some CPU usage cost associated with more calls but its not much and its just not worth using so much memory for streaming purposes.

This seems like an article that could have just been summed up and rewritten as the title. It's a conscious choice and consistent. I have no idea what use case would need to use 2GB per read call, if anything the limit seems excessive.

Does not reproduce :)

    // Assuming 2GiB length
    using var file = File.OpenRead("verylargefile.txt");

    var read = 0;
    var iterations = 0;
    var buffer = new byte[Array.MaxLength].AsMemory();

    while ((read = await file.ReadAsync(buffer)) > 0)
    {
        buffer = buffer[read..];
        iterations++;
    }

    Console.WriteLine($"Read {Array.MaxLength} in {iterations} call(s).");

Isn't GiB = 10^9, GB = 1024^3? Some people seem to use them interchangeably.

The other way around, author uses them correctly.

k/M/G/... come from SI which is based on 10^3. Ki/Mi/Gi/... are for 1024.

Other way around, 10^9 bytes is a gigabyte (GB) and 1024^3 is a gibibyte (GiB).

I have yet to hear anyone say "kibi" or "gibi" that wasn't mocking the term.

I believe the virtuous answer is that bytes are measured in powers of two, & hard drive manufacturers are dishonest. (Bits are measured in powers of ten)

I say “mibi” and “gibi” all the time, because my system denotes them as such and it sounds much cuter

> hard drive manufacturers are dishonest. (Bits are measured in powers of ten)

You'd have to add ISPs to that too, as network links are typically measured in decimal increments.

Yeah sure and it's completely irrelevant in this context and also the other way around

A good comparison. I couldn’t see that the author has created an issue on the golang GitHub. I assume that this would be considered a bug and that the golang community would be open to a fix.

Or not, because it simplifies code, no differences between platforms. And if syscall overhead once per each GB is really a problem you should probably try mmap or some other IO style.

Mmap seems specific to POSIX according to this post[1], so it's not a portable way of using memory mapped files. Since there are already differences between platform capabilities in golang there really shouldn't need to be an artificial limit to how large of a chunk you can read in one operation. In 20 years I'm sure 1 GiB will be considered small.

1: https://stackoverflow.com/questions/52589949/undefined-sysca...

The author saw performance issues with this restriction, which is why they wrote the post.

I would be curious to see the numbers on that.

Hello, author here. Just to clarify, I wasn't saying that this has any perf impact per se. I was trying to see why syscall showed up quite a bit in the profile, when reading a large file, no matter what buffer size I pass in. Rather than spend time finishing the actual work, I spent way too much time trying to understand the reason for the limit.

You should probably ask the author then. There are a bunch of links to various socials on the bottom of the post.

Only that there is no official mmap support in go, due to gc concerns, afaik

Hm? https://pkg.go.dev/golang.org/x/sys/unix#Mmap

Reply on news.ycombinator.com