Latency Numbers Every Programmer Should Know

Regarding the 90-day window, you are spot on. I never realized that until now. I made a mistake with the month, it should have been July 25 not June 25, so it came out after 60 days, not 90 days as I intended.

That's evidence #1236577 that I have no clue at all!

Some assorted facts:

Rotational latency -- The most common hard drive models are 7200 RPM. That is 120 rotations per second, or 1 rotation per 8.33 milliseconds. So, all things being equal, the average rotational latency will be half of that (best case: 0, worse case: 8.33 ms).

Seek time -- The most common hard drive models have 1 head on an actuator (though multi-actuator models are being developed). The actuator has to physical move the head across the surface of the platters so that it's over the bits you want. The worst case seek time is around 8ms on a typical hard drive, for an average of 4ms. The less random your workload is, that is, the smaller subset of the disk you are touching, the smaller your seek time will be.

IO size -- the amount of data you can read in a single hard drive rotation actually depends on where on the platter it is. If it's on the outer diameter of the drive, it might be e.g. 2MB, but on the inside diameter, only 1MB. For the reason you can actually get significantly higher throughput when reading from the outside diameter (typically the lower LBAs). Once you start doing even larger IOs, you're talking about waiting for multiple rotations for the IO to complete, which equals more time.

Read/Write distribution -- Hard drives have buffers on them to absorb writes, and it is asynchronously flushed in a very efficient manner (either when the hard drive is not doing anything, or is on the way to do another IO and happens to pass over the right part of the disk).

Queue depth -- Hard drives allow you to enqueue multiple concurrent IOs (most commonly from different threads/processes). SATA allows for a maximum of 32 concurrent IOs, whereas NVMe goes all the way up to 65K. Usually with an HDD though, you'll be on SATA. The number of concurrent IOs is referred to as the queue depth. You can improve your average latency by increasing queue depth, at the cost of peak latency. The reason for this is that hard drives will re-order queued IOs to reduce rotational latency and seek time. For example, imagine you issue 4 concurrent reads to LBAs 0, 10, 2, 5. The drive will likely reorder this to 0, 2, 5, 10 so that it doesn't have to spend as much time seeking. This will reduce the average latency, but the at particularly high queue depths, a particularly unlucky IO might get deprioritized for 100+ ms. If you care a lot about peak latency, bound your queue depth to 4 or so.

Highly recommended!

We've got all of our data going back years in AWS Athena, we're just waiting to have time to do something fun with it.

* 1MB: 6.52ms vs 2.16ms (3x speedup)

* 1GB: 5.73s vs 2.25s (2.5x speedup)

Tested with this C code, called as "bpredict-time [size] [sort]":

    #include <stdio.h>
    #include <stdint.h>
    #include <stdlib.h>
    #include <sys/time.h>
    
    #define SIZE (1024*1024)
    
    int cmp (const void *val1,
             const void *val2)
    {
        return *(uint8_t*)val1-*(uint8_t*)val2;
    }
        
    int
    main (int argc,
          char **argv)
    {
        uint8_t *array;
        size_t size = SIZE;
        struct timeval start, end;
    
        if (argc > 1) {
            size = atoi(argv[1]);
        }
        
        array = malloc(sizeof(*array) * size);
        if (array == NULL) { return -1;}
    
        for (size_t i=0; i<size; i++) {
            array[i] = random();
        }
    
        if (argc > 2) {
            qsort(array, size, sizeof(*array), cmp);
        }
    
        (void)gettimeofday(&start, NULL);
        int cnt = 0;
        for (size_t i=0; i<size; i++) {
            if (array[i] < 128) { cnt++;}
        }
        (void)gettimeofday(&end, NULL);
    
        
        printf("N ints < 128: %d\n", cnt);
        printf("Time: %f\n", (float)(end.tv_sec-start.tv_sec)\
            +(float)(end.tv_usec-start.tv_usec)/1000000);
        free(array);
        
        return 0;
    }

  There are 5001834 entries out of 10000000 less than 128
  Exeuction took 65335182 nanoseconds
  There are 5001834 entries out of 10000000 less than 128
  Exeuction took 26693952 nanoseconds

  #include <stdio.h>
  #include <stdlib.h>
  #include <time.h>
  #include <sys/types.h>
  
  #define DATASIZE 10000000
  #define NSEC_PER_SEC 1000000000
  
  int cmp(const void* a, const void* b)
  {
  	return *(unsigned char*)a - *(unsigned char*)b;
  }
  
  int countsmall(unsigned char* data)
  {
  	struct timespec start;
  	struct timespec end;
  	u_int64_t delta;
  	int lcv;
  	int numsmall;
  
  	numsmall = 0;
  	clock_gettime(CLOCK_MONOTONIC, &start);
  	for ( lcv = 0; lcv < DATASIZE; lcv++ )
  		if ( data[lcv] < 128 )
  			numsmall++;
  	clock_gettime(CLOCK_MONOTONIC, &end);
  
  	delta = (end.tv_sec   * NSEC_PER_SEC + end.tv_nsec) - 
  		(start.tv_sec * NSEC_PER_SEC + start.tv_nsec);
  			
  	printf("There are %d entries out of %d less than 128\n", numsmall, DATASIZE);
  	printf("Exeuction took %ld nanoseconds\n", delta);
  
  	return 0;
  }
  
  int main(int argc, char** argv)
  {
  	unsigned char data[DATASIZE];
  	int lcv;
  	srandom(time(NULL));
  	
  	for ( lcv = 0; lcv < DATASIZE; lcv++ )
  		data[lcv] = random();
  
  	countsmall(data);
  
  	qsort(data, DATASIZE, sizeof(unsigned char), cmp);
  
  	countsmall(data);
  
  	return 0;
  }

https://stackoverflow.com/questions/11227809/why-is-it-faste...

http://www.ibiblio.org/harris/500milemail.html

I have to say I am disappointed/surprised by how limited those savings are.

But yes the rest of your point is accurate.

https://www.nature.com/articles/nphoton.2013.45?error=cookie...

Surely, the dominant part of the path city to city is via fiber, so laser light?

That doesn't mean that's were most of the time goes (but ideally it should) - for latency things like light-to-signal, buffers, switching, carrier grade nat, all takes its toll.

Would be interesting to see a detailed breakdown of typical data center(... Cell phone, home broadband user) to same, though. Including every "point" in the path from ram, via cache, pci, network card, routers and switches, media converters and up again... For bonus something like USB keyboard touch to glyph on screen over udp or other low latency protocol...

This also implies that you could materially change that rtt number if you used a faster medium - e.g via air by way of microwave towers.

But even if this is assumed to be the time to transfer a UDP packet from one network card buffer to another network card buffer directly connected by a cable this seem extremely low.

CPU and data intensive heavy lifting is rarely done in such programs, it is delegated either to specialised libraries or some middleware in the form of a RDBMS. Most of these programs spend most of their time waiting for some IO event, so the few microseconds gained from the vector with a few hundred elements are negligible.

> But you can read a file from beginning to end.

That's what most programs actually do most of the time because files are essentially a stream abstraction. Programs that jump around a file would map it into memory, then the CPU and the kernel would do their best to cache the hot regions, even if the access to these regions is temporally or spatially distant.

third fault, which to me should ban this page from the internet: he has dared to put a "2018" text on his page, insinuating that it's new data, or some new insight, as opposed to being 15+ years old.

when the original text is more understandable than your visualization, you dun goofed