vlowther

Can't share my references with you directly, the implementation I wrote is closed-source and is heavily intermingled with other internal bits. But I can provide examples:

1. syscall.Iovec is a struct that the writev() systemcall uses. You build it up something like this:

    func b2iov(bs [][]byte) []syscall.Iovec {
        res := []syscall.Iovec{}
        for i := range bs {
            res = append(res, syscall.Iovec{Base: &bs[i][0], Len: uint64(len(bs[i])}
        }
        return res
    }

Then, once you are ready to write:

    func write(fi *os.File, iov []syscall.Iovec, at int64) (written int64, err error) {
        if _, err = fi.Seek(at, io.SeekStart); err != nil {
            return
        }
        wr, _, errno := syscall.Syscall(syscall.SYS_WRITEV, fi.Fd(), uintptr(unsafe.Pointer(&iov[0])), uintptr(len(iov)))
        if errno != 0 {
            err = errno
            return
        }
        written = int64(wr)
        err = fi.Sync()
        return
    }

These are not tested and omit some more advanced error checking, but the basic idea is that you use the writev() system call (POSIX standard, so if you want to target Windows you will need to find its equivalent) to do the heavy lifting of writing a bunch of byte buffers as a single unit to the backing file at a known location.

2. Yeah, I just zero-filled a new file using the fallocate as well.

3. I handled max buffer age by feeding writes to the WAL using a channel, then the main reader loop for that channel select on both the main channel and a time.Timer.C channel. Get clever with the Reset() method on that timer and you can implement whatever timeout scheme you like.

4. No, it is not needed, but my WAL implementation boiled down to a bunch of byte buffers protected by a rolling CRC64, and for me just mmap'ing the whole file into a big slice and sanity-checking the rolling crcs along with other metadata was easier and faster that way.

Having written one of these, a few optimizations will go a long way:

1. syscall.Iovec allows you to build up multiple batches semi independently and then write them all in a single syscall and sync the file with the next one. It is a good basis for allowing multiple pending writes to proceed in independent go routines and have another one have all the responsibility for flushing data.

2. It is better to use larger preallocated files than a bunch of smaller ones, along with batching, fixed size headers and padding write blocks to a known size. 16 megabytes per wal and a 128 byte padding worked well for me.

3. Batching writes until they reach a max buffer size and/or a max buffer age can also massively increase throughput. 1 megabyte max pending write or 50 ms time passed worked pretty well for me for batching and throughput to start with, then dynamically tuning the last bound to the rolling average of the time the last 16 write+sync operations (and a hard upper bound to deal with 99th percentile latency badness) worked better. Bounded channels and a little clever math makes parallelizing all of this pretty seamless.

4. Mmap'ing the wals makes consistency checking and byte level fiddling much easier on replay. No need to seek or use a buffered reader, just use slice math and copy() or append() to pull out what you need.

It is not hard research, it is "just" a lot of plain old boring engineering.

> Does iPXE have a ca-certificates bundle built-in, is there PKI with which to validate kernels and initrds retrieved over the network at boot time

For HTTPS booting, yes.

> how does SecureBoot work with iPXE?

It doesn't, unless you manage to get your iPXE (along with everything else in the chain of control) signed.

Oooh. 6809 resurrection project. I studied that arch obsessively when I was a kid with a Tandy Color Computer.

One of the great thinkers of the modern era. He will be missed.

Terrorists willing to blow themselves up at a time of their choosing is the key there. With nuclear waste hot enough to be a menace, the window between getting the material and being painfully incapacitated and/or dead by the material is pretty damn small. Especially since nuclear plants tend to have things like lots of checkpoints and security.

More than that. If you have ever had to fix a bug in code common to multiple maintained releases of a project, being able to apply the same patch to them all as its own thing instead of having multiple cherry-picked commits with identical content would be nice.

That is a distinction that only exists because a Git repo is an ordered sequence of snapshots of your code that get translated to and from patches for human consumption. As far as I can tell, a Pijul repo is more of a giant dependency graph of patches that when merged together form a consistent snapshot of your code.

Charge, mass, velocity, and spin are objective. Value exists only in the eye of the beholder.

Fish Speakers. God Emperor Of Dune.

Nothing. You have to get into INTERCAL territory to find something more tedious to work with than COBOL.

Anything that implements or consumes `io.Reader` or `io.Writer` would dispute that.

Why the extra "clone the code" step in the Go developer experience? Been using Go since 1.0, never had to do that.

> The planet will die off first.

Definitely not. It may go through a bottleneck in which humanity has to adapt or die (not for the first time, I might add), but (if anything) after the climate has stabilized and life has adapted it will be more fecund than anything -- all those new shallow seas will be totally awesome biomes.

Probably Go. It is really well suited to this sort of thing.

> Good luck getting that to run in different environments because static or not, nearly every executable has dependencies.

Go compiled with CGO_ENABLED=0 disagrees, especially since embedding was folded into the standard library.

> Which of course you'll need to setup, patch, figure out how to restart apps when they go down and on and on. There's no easy out.

Recompile, upload, restart. If the VPS server goes down, that is what their tech support is for.

> And how do you back that up?

Either use LVM snapshots and tar or sigstop/tar/sigcont, and rely on whatever embedded storage engine you are using to be able to recover from what will appear to be a bog standard recovery from unexpected shutdown. If you are nice, arrange for an API to be able to take a complete snapshot of your running data.

> Postgres, Docker, Infra as Code and Kubernetes are pretty much best practices on any development.

That is debatable. There is a whole lot to be said about the virtues of a single binary that has no dependencies beyond a vaguely modern Linux distro.

BBEdit is my weapon of choice for reading massive log dumps and database dumps for support at $DAYJOB. It does not seem to ever choke no matter how many hundreds of megabytes if text I need to plow through.

... And nothing of value was lost.

Covid pandemic proved that most office jobs can be done remotely with little to no loss of productivity.

Because I would rather write one generic container library than n slightly different ones for each data type of deal with screwing around with casting interfaces?

Yes.

well, FAT16/FAT32/EXT2 are not journalled, so making them lose data in a giant variety of ways is a trivial affair. You might want a higher bar than that.

Not that I have seen. map[T]struct{} is more common, with if _,ok := m[x]; ok {} used for testing if a key is in the map. The space savings really add up with large sets.

Instructions unclear, now suffering from waterfowl infestation. I think some of them are geese.

You can now. That was not always the cas e, and that is one of many reasons I used postfix instead of sendmail back in the day.

Ethanol is awesome when it comes to knock resistance because it is much slower burning than gasoline is with all other parameters being equal. Where it bites is that it requires better seals than gasoline, and it has lower energy density, which means you need higher flow fuel pumps, tubing, filters, and injectors to make the same power.

Do that, though, and you can run a turbocharged engine at much higher levels of boost and create much more power than you would be able to do with anything short of race gas or even more exotic and dangerous fuels.

Good old nuclear salt water rockets: https://www.npl.washington.edu/av/altvw56.html

Just don't point the exhaust at anything inhabited.

Given that your squishy brainmeats also consist of neural networks that have been trained directly by years of experience and indirectly by billions of years of evolution, I can write off your response as just a weighted average retrieval across a compressed dataset of trillions of your experiences.

Twenty years ago it would have been the Scary Devil Monastery. But that was another lifetime, and besides, the protocol is dead.