codys

Yes, but I suspect there might be some confusion by the author and others between "io_uring completion of a write" (ie: io_uring sends its completion queue event that corresponds to a previous submission queue event) and "fsync completion" (as you've put as "completion of all writes", though note that fsync the api is fd scoped and the io_uring operation for fsync has file range support).

The CQEs on a write indicate something different compared to the CQE of an fsync operation on that same range.

> > you lose the durability guarantee that makes databases useful. ... the data might still be sitting in kernel buffers, not yet written to stable storage.

> No! That's because you stopped using fsync. It's nothing to do with your code being async.

From that section, it sounds like OP was tossing data into the io_uring submition queue and calling it "done" at that point (ie: not waiting for the io_uring completion queue to have the completion indicated). So yes, fsync is needed, but they weren't even waiting for the kernel to start the write before indicating success.

I think to some extent things have been confused because io_uring has a completion concept, but OP also has a separate completion concept in their dual wal design (where the second WAL they call the "completion" WAL).

But I'm not sure if OP really took away the right understanding from their issues with ignoring io_uring completions, as they then create a 5 step procedure that adds one check for an io_uring completion, but still omits another.

> 1. Write intent record (async)

> 2. Perform operation in memory

> 3. Write completion record (async)

> 4. Wait for the completion record to be written to the WAL

> 5. Return success to client

Note the lack of waiting for the io_uring completion of the intent record (and yes, there's still not any reference to fsync or alternates, which is also wrong). There is no ordering guarantee between independent io_urings (OP states they're using separate io_uring instances for each WAL), and even in the same io_uring there is limited ordering around completions (IOSQE_IO_LINK exists, but doesn't allow traversing submission boundaries, so won't work here because OP submits the work a separate times. They'd need to use IOSQE_IO_DRAIN which seems like it would effectively serialize their writes. which is why It seems like OP would need to actually wait for completion of the intent write).

> But database engines are absolutely the target of io_uring's feature set and they're expected to be managing this complexity.

io_uring includes an fsync opcode (with range support). When folks talk about fsync generally here, they're not saying the io_uring is unusable, they're saying that they'd expect the fsync to be used whether it's via the io_uring opcode, the system call, or some other mechanism yet to be created.

> Yes, you do need to check if both records are written and then report it back to the client. But that is a non-fsync request and does not tax your system the same as fsync writes.

What mechanism can be used to check that the writes are complete if not fsync (or adjacent fdatasync)? What specific io_uring operation or system call?

They do not state that it is exclusively collected for those purposes, only that those purposes is included. As written, they'd be in line with their policy to collect that data for any purpose (including those listed).

this seems possible to avoid as an issue without needing IP certs by having the configuration supply both an IP and a hostname, with the hostname used for the TLS validation.

Were there any Microsoft XP security issues caused by "Easter eggs" prior to that policy change? Or was this just put in place as a policy because it was easy to put in place?

I agree that there are problems when laying multiple build systems on top of one another, and I see that often as a user of nix (it's also bad with rust projects that use cargo, and though there are a variety of options folks have written they all have tradeoffs).

To some extent, the issue here is caused by just what I was discussing above: Nix derivations can't dynamically add additional derivations (ie: build steps not being able to dynamically add additional build steps makes things non-optimal).

I am hopeful that Nix's work on dynamic derivations will improve the situation for nix (with respect to bazel, cargo, and others) over time, and I am hopeful that other build systems will recognize how useful dynamically adding build steps can be.

True. That isn't something the compiler can do.

That said, that (determining the c flags and ld flags for dependencies) is something that might be able to be mixed into compilation a bit more than it is now. Could imagine that if we annotate which compilation units need a particular system library, we could start building code that doesn't depend on that library while determining the library location/flags (ie: running pkg-config or doing other funny business) at the same time.

Or since we're in the connected era, perhaps we're downloading the library we require if it's not found and building it as an embedded component.

With that type of design, it becomes more clear why moving as much to the build stage (where we can better parallelize work because most of the work is in that stage) and more accurately describing dependencies (so we don't block work that could run sooner) can be helpful in builds.

Doing that type of thing requires a build system that is more flexible though: we really would need to have the pieces of "work" run by the build system be able to add additional work that is scheduled by the build system dynamically. I'm not sure there are many build systems that support this.

yep. C's really come a long way with the special operators for checking if attributes exist, if builtins exist, if headers exist, etc.

Covers a very large part of what is needed, making fewer and fewer things need to end up in configure scripts. I think most of what's left is checking for items (types, functions) existence and their shape, as you were doing :). I can dream about getting a nice special operator to check for fields/functions, would let us remove even more from configure time, but I suspect we won't because that requires type resolution and none of the existing special operators do that.

I did something like the system described in this article a few years back. [1]

Instead of splitting the "configure" and "make" steps though, I chose to instead fold much of the "configure" step into the "make".

To clarify, this article describes a system where `./configure` runs a bunch of compilations in parallel, then `make` does stuff depending on those compilations.

If one is willing to restrict what the configure can detect/do to writing to header files (rather than affecting variables examined/used in a Makefile), then instead one can have `./configure` generate a `Makefile` (or in my case, a ninja file), and then have the "run the compiler to see what defines to set" and "run compiler to build the executable" can be run in a single `make` or `ninja` invocation.

The simple way here results in _almost_ the same behavior: all the "configure"-like stuff running and then all the "build" stuff running. But if one is a bit more careful/clever and doesn't depend on the entire "config.h" for every "<real source>.c" compilation, then one can start to interleave the work perceived as "configuration" with that seen as "build". (I did not get that fancy)

[1]: https://github.com/codyps/cninja/tree/master/config_h

On audio sync issues, I unfortunately see them with jellyfin rather frequently on apple tv when using homepods as audio output. I end up having to enable the "native player" in the experimental settings to get the audio in-sync.

I've previously reported this to the developers of the app, and they've closed the issue saying it was a bug in one of their dependencies, without fixing the issue. It remains unfixed.

This is a fundamental misunderstanding of the structure of the linux kernel, the nature of kernels in general, and the ways one performs automated verification of computer code.

Automated verification (including as done by rust in it's compiler) does not involve anything popularly known as AI, and automated verification as it exists today is more complete for rust than for any other system (because no other widely used language today places the information needed for verification into the language itself, which results rust code being widely analyzable for safety).

Human validation is insufficient and error prone, which is why automated verification of code is something developers have been seeking and working on for a long time (before rust, even).

Having "explicit" (manual?) memory management is not a benefit to enabling verification either by humans or by machines. Neither is using a low level language which does not specify enough detail in the type system to perform verification.

Kernel modules aren't that special. One can put a _lot_ of code in them, that can do effectively anything (other than early boot bits, because they aren't loaded yet). Kernel modules exist for distribution reasons, and do not define any strict boundary.

If we're talking about out-of-tree kernel modules, those are not something the tend to exist for a long time. The only real examples today of long lived out-of-tree modules are zfs (filesystem) and nvidia (gpu driver). These only exist out-of-tree because of licensing and secrecy. This is because getting code in-tree generally helps keep code up to date with less effort from everyone involved: the people already making in-tree changes can see how certain APIs are being used and if those in-tree folks are more familiar with the API they can/may improve the now-merged code. And the formerly out-of-tree folks don't have to run their own release process, don't have to deal with constant compatibility issues as kernel APIs change, etc.

If it was a good approach, the C folks would have copied around C files instead of having common core code. They did not do that though, because it'd be counter productive due to increasing maintenance burden.

Everyone knows it would increase maintenance burden, decrease reliability, and increase the amount of apparent churn of rust code in the linux kernel.

It's not charitable to construct a contrived situation no one is talking about and place that into the mouth of a commenter.

The commenter appears to have been referring to a specific law in the netherlands without stating this.

The original Firefox sync worked like this (with a unique code and pairing instead of an explicit account) (this is so on the nose I suspect you may know this).

This blog post goes over some of that history: https://blog.mozilla.org/services/2014/04/30/firefox-syncs-n...

> Finally, as I imagine it, mixing 32- and 64-bit time_t ABIs would be prohibited — implicitly, as 64-bit time_t symbols aren't ever visible to 32-bit time_t code and vice versa — with code that needs to support non-native time_t values (for I/O, IPC, etc.) left its own devices, just like code dealing with, e.g., non-native integer and floating-point formats today.

It's pretty easy to imagine having both 32-bit time_t and 64-bit time_t in a single "executable" as long as the interfaces actually in use between 32-bit-time and 64-bit-time components don't use `time_t` (or derived types).

iow: if the fact that `time_t` is 32-bits is kept entirely internal to some library A used by some library B (by virtue of library A not having any exposed types with time_t that are used by library B, and library A not having any functions that accept time_t or derived types), there's nothing preventing mixing 32-bit-time_t code and 64-bit-time_t code in a single executable/process (in this theoretical case where we use a LFS (ala _FILE_OFFSET_BITS, etc) for time_t).

LFS had/has the same capability for mixing (with off_t being the type in question there).

Presumably one wouldn't throw away the built packages if some other build fails just because we're delaying installing the packages to `/` until later.

Huh. If it's OK to have instructions like those for users to run through, then I guess I don't understand what's holding up switching to 64-bit time_t for gentoo. Seems like they could make a new profile and tell folks to deal with it. Perhaps I don't understand what the requirements are here, which might indicate that the requirements aren't clear for others too.

Before getting into the details here about why we need to read `libA.so` at build time: Perhaps the documentation gentoo provides about `ROOT` that was noted by rini17 is incorrect/misleading and `ROOT` is used as the source of build time dependencies automatically and the documentation was intended for ebuild writers and not emerge users? (in which case, Gentoo should fix/clarify it).

"link" has 2 meanings:

1. As you've pointed at, at runtime we have linking occur that reads files.

2. Additionally, building software with `gcc`/etc, we tell it what libraries it needs to link against (`-lA`). When we do that, `gcc` as part of the build process searches for `libA.so` (in various locations dependent on its configuration and arguments), reads the `libA.so` file, and uses that info to determine what to emit in the object file that `gcc` is building. So we need to read a shared library. The same is true for headers used at build time. How much this actually ends up mattering varies depending on architectural details, and whether libraries have different symbols exposed (or generate different header files) depending on what the bit-width of time_t is. It may well be that for most packages we could have the build-time linking occur against a 32-bit time_t object as long as runtime linking is done to the 64-bit time_t object (assuming one is using identical package versions or at least versions with a compatible ABI).

Fat binaries (packing together 2 entirely different objects and picking one of them) could potentially work here, though I suspect we'd run into issues of mixed 32-bit/64-bit time_t things.

Another option that's closer to how things work with LFS (large file support) (mostly in the past now) is to use create different interfaces to support 64-bit time_t, and pick defaults for time_t at compile time with a macro that picks the right impl.

Also possible could be having something like a version script (https://sourceware.org/binutils/docs/ld/VERSION.html) to tell the linker what symbols to use when 64-bit time_t was enabled. While this one might have some benefits, generally folks avoid using version scripts when possible, and would require changes in glibc, making it unlikely.

Both of those options (version script extention, and LFS-like pattern) could allow re-using the same binary (ie: smaller file size, no need to build code twice in general), and potentially enable mixing 32-bit time_t and 64-bit time_t code together in a single executable (not desirable, but does remove weird link issues).

No, what rini17 pointed out is a problem here, and I think my previous comments that setting ROOT might be "part of a solution" are less accurate (as it seems `ROOT` is even less of a complete solution here).

Lets consider this hypothetical representation of the 64-bit time issue:

In `ROOT=/`, we have a `libA.so` with 32-bit time_t, and a `libB.so` with 32-bit time_t that depends on `libA` (ie: `libB.so` -> `libA.so`).

We rebuild package `A` with a special root (`ROOT=/tmp/new`) which has 64-bit time_t. We now have a `/tmp/new` with `libA.so` built for 64-bit time_t. We now want to build `B` for `ROOT=/tmp/new` with 64-bit time_t and have it link against the new `/tmp/new/lib/libA.so`.

But setting `ROOT=/tmp/new` doesn't cause us to use the `/tmp/new` to as our source to link against things. So (as it currently stands) we'll end up trying to link `libB.so` (64-bit time_t) against `/lib/libA.so` (32-bit time_t), and things just won't work.

If gentoo changes/extends ROOT (or adds something else like ROOT but used to locate dependencies for linking), then it could be a piece of a solution. But it sounds like it's pretty far from that in intention/practice right now.

Simply changing `ROOT` doesn't magically enable us to have `emerge` update our normal `ROOT=/`. There's extra steps here (the syncing, the bind mounting, etc) for the `emerge` with a custom `ROOT` to do an update to `/` itself.

And that's if we have a `ROOT` that allows us to source dependencies (libraries to link against, includes, etc). Currently it's unclear to me how exactly `ROOT` works, probably meaning there needs to be a bit more documentation there and potentially indicating that it might not be used much right now.

It's true that Gentoo has some pieces of what they need to build an implementation of option 1 (build packages that depend on other packages before completeing an install), but it's currently the case that that is not what happens when one runs `emerge` (the gentoo packaging build/installing tool), as you've noted one would need to write scripts that wrap emerge (or do the same work manually) to attempt to accomplish this today.

I suspect that using bind mounts and overlays (to allow the "building packages" chroot a view of the "installed" root) could be used to accomplish this, or alternately some filesystem snapshotting features if we're thinking about this from the "external to emerge" angle. (It's my understanding that for Gentoo to do this ABI change, though, they need some solution integrated into emerge).

To some extent, this kind of potential model also reminds me of systems that integrate their package updating with filesystem snapshotting to allow rollbacks and actually-atomic upgrades of many files. I think one or more of the solaris distributions did this?

nixos is an example of a distro that is source based and can do the atomic rebuild here because it has a way to build packages against other packages that aren't "installed". In nixos, this is because there are very few things that get "installed" as the global, only thing to use. But one could imagine that Gentoo could build something that would allow them to at least build up one set of new packages without installing them and then install the files all at once.

Yep, agreed. Though it does expose an option here which would be "have glibc provide a mechanism for other libraries (likely more core/widely used libs) to support both ABIs simultaneously.

Presumably if that had been done in glibc when 64-bit time_t support was added, we could have had multi-size-time ABI support in things like zlib by now. Seems like a mistake on glibc's part not to create that initially (years ago).

Though if other distros have already switched, I'd posit that perhaps Gentoo needs to rethink its design a bit so it doesn't run into this issue instead.

Nixos has it easier here because they don't require packages to be "installed" before building code against them. For Gentoo, none of their build scripts (ebuilds) are written to support that. It's plausible that they might change the embuild machinery so that this kind of build (against non-installed packages) could work, but it would need investigation and might be a difficult lift to get it working for all packages.

"treat it as a cross compilation scenario" is essentially what the post discusses when they mention "use a different CHOST". A CHOST is a unique name identifying a system configuration, like "x86_64-unknown-linux-gnu" (etc). Gentoo treats building for different CHOSTs as cross compiling.

There are a few options for Gentoo not discussed in the post, possibly because for Gentoo they would be a larger amount of work due to the design of their system:

1. Allow building against packages without installing them. The core issue here is that Gentoo package build and install happen as a single step: one can't "build a bunch of things that depend on one another" and then "atomically install all the build items into place". This means that Gentoo can easily be partially broken when one is doing updates when an ABI change occurs (modulo so versioning, see next option). This issue with 64-bit time_t is an example of an ABI change that folks are very aware of and is very widespread. It's also an example of something that causes an ABI change that isn't handled by the normal `.so` versioning scheme (see next option).

2. Extend the normal `.so` versioning to capture changes to the ABI of packages caused by packages they depend on. Normally, every `.so` (shared object/library) embeds a version number within it, and is also installed with that version number in the file name (`libfoo.so.1.0.0`, for example, would be the real `.so` file, and would have a symlink from `libfoo.so` to tell the linker which `.so` to use). This shared object version is normally managed by the package itself internally (iow: every package decides on their ABI version number to enable them to track their own internal ABI breakages). This allows Gentoo to upgrade without breaking everything while an update is going on as long as every package manages their `.so` version perfectly correctly (not a given, but does help in many cases). There is a process in Gentoo to remove old `.so.x.y.z` files that are no longer used after an install completes. What we'd need to do to support 64-bit time_t is add another component to this version that can be controlled by the inherited ABI of dependencies of each `.so`. This is very similar in result to the "use a different libdir" option from the post, but while it has the potential to set things up to enable the same kinds of ABI changes to be made in the future, it's likely that fixing this would be more invasive than using a different libdir.

> there would be inconveniences like libc.so has 64-bit off_t

glibc specifically has support for the 32-bit and 64-bit time_t abi simultaneously.

From the post:

> What’s important here is that a single glibc build remains compatible with all three variants. However, libraries that use these types in their API are not.