Accidentally nonblocking

The 'nonblocking' here is just a symptom of a much larger problem: abstraction is a great way to build complex stuff out of simple parts but it is also a great way to introduce all kinds of effects that you weren't aiming for in the first place and this particular one is easier than most to catch. You can find the same kind of problems at all levels of software systems, all the way up to the top where dosomethingcomplex() and if it fails dosomethingcomplex() again is the cure.

Writing easy to understand code is a big key to solving this kind of problem, I've always tried (but probably never succeeded) in writing code in the simplest way possible, as soon as I find myself reaching for something clever I feel it is a mistake. Either circumstance (some idiot requirement, such as having to use the wrong tools for the job) or need may be used occasionally to transgress the rule but if you do it with any regularity at all (and without documenting the particular exception and the motivation to go outside of the advised lines) you are almost certainly going to regret it. (Or your successor may one day decide to pay you a house-call with a blunt object...)

When a socket polls for readiness in Unix, it does not mean that a subsequent read will succeed. The obvious case is when another thread reads from the socket before you do. A less obvious case is that some kernels, such as Linux, implement lazy checksum verification. Linux will wake up any waiting threads when a packet comes in (including marking an open file table entry as readable), but the checksum isn't verified until an actual read is attempted. If the checksum fails, the packet is silently discarded. If the socket wasn't in non-blocking mode, your application will stall until the next packet is received.

The JRE had (and maybe still has) a bug like this, where it assumed poll meant that a subsequent read was guaranteed to succeed or fail immediately.

This particular issue is less common today with checksum hardware offloading, but the correctness and robustness of your software probably shouldn't depend on particular network chipsets.

Another bug I've seen several times is assuming that a write to a UDP socket won't block. You can usually get away with this on Linux because the default buffers are so huge. As with the above issue, it really only shows when your application (and thus the network) is under significant load.

One conclusion I draw from this is that while people go to great lengths to implement a supposedly scalable architecture, most of the time developers never see the kinds of heavy load that such architectures are designed for. If they had, they would have discovered these sorts of issues. Fortunately or unfortunately for me, I discovered both of the above issues the hard way.

[1] If you're wondering why I kept writing "open file table entry" instead of descriptor, it's because they're not the same thing. And some day I expected a few CVEs to be issued related to overlooking such distinctions. For example, on BSDs /dev/fd/N duplicate a descriptor point to the same file table entry, just as dup(2) does. On Linux /dev/fd is a symlink to /proc/self/fd. /proc/self/fd creates a new file table entry. In the former case, software setting or unsetting O_NONBLOCK effects all other references to that entry.

If xterm were designed so that e.g. xevents() had only the responsibility of fetching bytes from the X socket and do_xevents() and everything else had only the responsibility of handling bytes from an buffer, there would be no temptation to poll in two different functions. Only one function would even know that the byte source is a socket; the rest just know about the buffer.

Yes please, I'd like that. The code that ends up on my desk would be easier to understand and refactor.

Instead of a proper fix, the developer changes the socket to nonblocking

Sometimes yes, and sometimes the developer decide to spawn a thread, and now you have lots of problems...

The client side grew up against serial ports ( yes, those are still a thing ) , where you don't have this problem. The owner of the client side was more or less in incredulous terror when I broached this subject. Sigh. So I just ignored them. Big sigh. If it failed, there were retries so the only cost was a little delay now and again.

You cannot fragment UDP unless you're prepared to add some method of sequencing as part of the application protocol. Each UDP PDU needs to be fully atomic otherwise.

For cases analogous to SNMP row creation ( in which multiple varbinds[1] determine the outcome ), there is the "as if simultaneous" rule as a heuristic - all PDUs related to creating a row must be cached and only applied when the row state is set. And sometimes you can configure things to send all varbinds in one PDU.

[1] a varbind is a triple of the set/get/next/multi operator, the object id and if applicable a value as encoded by the SNMP Basic Encoding Rules.

So your little serialization protocol? It suffers all the heartache of a full-on transactional database processing system.

These things are this way because communications are like that.

The funny thing is that writing nonblocking code doesn't have to be as hard as it is, you just have to get into the mindset. It's easy to say "well, I have to have the result coming back from my JDBC/REST/etc call before I know what to do with it" and that's not the case at all, especially when you're working in a strongly-typed environment.

Now you have the problem when trying to combine multiple such libraries. For example, you can try using GTK+ and QT in an application at the same time.

One thing that has always bugged me is that there is a lack of standardized (cross platform) non-blocking DNS lookup functionality. This adds to the main-loop complexity, since you have to poll() certain types of resources, and have to deal with threads or subprocessed in order to look for DNS results.

Well written frameworks like Qt abstract away this complexity, but that may not always play nice when mixing libraries.

That only happens if somehow you have received data that is EXACTLY the size of the buffer you send to recvfrom. If you have read less, you know that there is no need to call it again. If your buffer is full, then odds are there is more data to read so the next call won't be useless.

I suspect you actually had an extremely small proportion of "extraneous" calls.

    void
    xevents(void)
    {
        if (poll() || poll())
             while (poll()) {
             /* ... */
             }
    }

Without ES6 and Babel, what's the way to write if statements with async?

  if (x) {
    _afterX(null, x);
  } else {
    getX(_afterX);
  }
  function _afterX(err, x) {
    if (err) return;
    // use x
  }
  // do stuff that doesn't depend on x