quotemstr

Interesting they'd recommend Decentraleyes: its default configuration disabled all HTTP link prefetching, which hurts performance in a way not apparent from the extension description.

Sure, but you have to take pains to make sure that journalctl -f doesn't skip events or print some twice. It can be made to work, for sure, but it just seems easier to print logs via a daemon instead, especially because if you go through a deamon, you turn the disk format from an interface into an implementation detail.

So write amplification on btrfs doesn't matter?

So, yes, journald does in fact do bulk copies of log files on rotate. btrfs is hardly some fringe FS and its COW-flag behavior is documented and well-known. I'd expect extensive work on journald's storage engine to have uncovered this behavior at some point.

> When did this become a discussion limited to journald on btrfs?

btrfs is in the HN thread title.

> and that seems like something btrfs should fix at some point

Amazing. The Linux kernel should change to work around journald's inflexibility?

What someone should fix at some point is journald's strange IO patterns and hard-coded "helpful" attribute changes. I'd rather it just rename the file and let me do any defrag/compression/flag-setting I want than do anything with chattr behind my back in ways I can't even configure.

> it's only used in a workaround when clearing FS_NOCOW_FL fails.

Clearing FS_NOCOW_FL doesn't work on btrfs for non-empty files. So what do you think journald is doing when it notices that it can't clear the flag?

journald does do a rewrite of the log file on rotation, so you're paying that IO anyway even if you ignore the dumb hash table updates.

https://github.com/systemd/systemd/blob/8f4cd7de43d1e6e94687...

WAL writeback is at least principled and efficient. It works out to being equivalent to the custom Parquet-rotation things others mention, but already implemented and working.

So, yes, WAL for logs, because LSM is the design everyone converges on and a WAL is LSM. Better to use the LSM already implemented and debugged in a database than write some random new one in terms of Parquet that's going to have to do the same stuff in the end anyway, just with novel bugs and no tool support.

(And look, I don't give a damn what "DB" people say, a WAL writing back to a DB IS log... structured... merge under any fucking sensible definition of what LSM means.)

How do readers get log entries that haven't made their way into one of the parquet archive files? If the tip is some kind of live-update DB, that DB has to support concurrent readers who can't block writers. Or would you just make log messages invisible to readers until they made their way into a stable Parquet file?

Forget about DB terminology and look at what's happening ON THE DISK. ON THE DISK, is what DuckDB doing any less efficient than what your custom Parquet thing would be doing?

SQLite has the problem of a malicious reader being able to hold up writers. Maybe that's fine in most cases, but in a system log, I don't think that's acceptable. IMHO, options are to indirect through a daemon anyway (e.g. using Quack) or do a lot of engineering to make it possible for an unprivileged reader to open() the log file and read it in such a way that it can't interfere with privileged writers.

Your LSM compaction strategy is going to have to solve the same problem anyway, isn't it? DuckDB is an LSM compaction strategy of this form, already done.

> I'd consider any btrfs install that does not have nocow on...to be defective.

You're getting COW on the extents if you're snapshotting anyway.

Once you solve enough problems using raw Parquet or Feather or whatever and you end up with something that looks like a DB anyway, so you might as well use a DB.

Uh, who said anything about writing 122880 rows every time you do a single insert into DuckDB? There's a WAL. Consolidation happens in big chunks. (And it's not like journald log rotation is somehow better than WAL consolidation.)

We shouldn't be making momentus choices of data format based on vague and incorrect understandings of data formats.

SQLite here is okay, but DuckDB or LevelDB would be better. Either way, no need to invent a new storage format.

Thank you for your work. ISTM the workload is naturally LSM-shaped.

> If you write a few bytes into some arbitrary position within a file, the storage has to write back the whole block, despite your only changing a tiny fraction of it. If those few bytes happened to cross a block boundary, guess what? two blocks get written.

Exactly. So either make the format append-only or make it append-mostly with occasional writebacks from the append-only log to the main data structure. Nice and simple.

> I'd expect the mmap aspect to be causing more/mispredicted reads, and polluting the page cache with unrelated contents (you tend to end up with the entire journal cached IIRC, if you have enough memory).

If you used an LSM or append-only approach, you could MADV_DONTNEED the pages behind your write cursor pretty easily.

Systemd should just use DuckDB. It's perfect for this job.

"But isn't it an OLAP database? Shouldn't you use SQLite for something that's vaguely real-time?"

Eh, in this instance, I think I'd prefer the columnar design and automatic compression DuckDB affords. Log entries have lots of little fields, many of which are unchanging from row-to-row, and DuckDB excels at storing this kind of data.

BTW: no, you don't need O(N*log(N) writes for DuckDB. No, you're not doing a whole block-group write for every message. No, Parquet is not a magical solution. I mean, maybe it's fine, but DuckDB is already columnar, and arguably better at it.

Seems like there are a lot of mistaken impressions about DB storage engines out there.

> Now every project has at least one perfectly made ASCII diagram - instant LLM tell.

Damn. Now my artist-mode in Emacs skills are useless.

Electron is just another god damn GUI toolkit! What's the fundamental difference between it and, say, GTK4? It's just fucking widgets!

My real problem with Electron is the multi-process render architecture, which I think is pointless for the "just a GUI toolkit" case of rendering only trusted content. I'm 100% fine with Slack, Spotify, whatever using Electron as their choice of GUI toolkits. These come and go. I'm miffed that they use multiple processes when there's no real security boundary between them just because there is a security boundary in the homologous scenario on the web.

This hack is 99% giving people the control over their own computers they should already have had. Guy is a Robin Hood.

Even physical DRAM access would be thwarted by transparent total memory encryption, so this hack is still something else.

This is the level of access the rightful owner of a computer should have to his own system.

He should also be able to fuse away this access forever, to be fair. But out of the box, when I get a new laptop, I should be able to read and write every byte of DRAM.

Information that enters the mind through the eyes and ears is nothing like a drug that alters consciousness at a low level. Words are just words. You don't get to control the words people choose to hear.

Controlling my information consumption is not a legitimate function of the state. If algorithmic feeds get a lot of use, it's because people like them. Is that so hard to comprehend? That your preferences are not universal? And that you shouldn't use the government to bludgeon people into accepting the kind of information feed you, personally, would prefer for them? What gives you the right to do so?

What you calling "addictive" is just revealed preferences of the populace.

It could also react to hitting the timeout with a hard reset. Annoying perhaps, but at least safe. Ancient principle of system design is that when you must fail, it is better to fail safe than fail deadly even when it's annoying in the short term.

ARM has EL3, which is basically the same thing. There's nothing inherently wrong with the CPU having multiple privilege levels. The problem with SMM has always been its user-hostile opaque implementation, not that the technical mechanism exists.

> The code waits for all cores to enter SMM, or for up to 1 second, whichever occurs first.

See, this is why the mantra that all blocking operations should have a timeout is stupid and short-sighted no matter how many times junior devs and AIs bleat it in code review. Continuing after arbitrary timeouts usually violates invariants, and failing after arbitrary timeouts introduces hard-to-debug failures under load.

Better for the system to hang so you can debug it --- and maybe reboot as a whole via a watchdog --- than for the code to say "Oh, this operation is supposed to be done after one second, but isn't. Situation normal, everything fine. We continue."

No. That situation is very much not fine.

Doesn't everyone do this now? It's hardly a new idea. Yet every time someone proposes the idea, people fawn over it and proclaim it the best thing ever.

Yes, you can inject tokens via a proxy. What else is new?

If you're relying on a prompt to constrain agent behavior, you've already lost.

Huh? You don't mention the built-in sandboxing options in things like Codex. Why do people pretend these features don't exist?

https://learn.chatgpt.com/docs/permissions

Another one of those interventions that'll just put US labs at a disadvantage. The public at large doesn't want to use models gimped so as to protect a bunch of pre-AI special interests who should adapt, not obstruct.

Just add async_uuid to your IDL locally and rerun MIDL. The server doesn't know or care that it's being called asynchronously. It's a pure client thing. If you're calling an IFoo in a different apartment, you can do the call asynchronously by changing your client configuration without the IFoo implementation having to be aware that anything is going on.

You can also use IRpcChannelBuffer3 to make the call asynchronously no matter what the IDL says. You'll have to write the proxy plumbing yourself, but it'll work, and you don't even need to rerun MIDL.

And COM enforces lifecycle rules, transparent remoting, uniform activation (constructors basically), security primitives, marshalling, transparent async calls (yes, you can call any method asynchronously), and a ton of other things. It's actually very good and it's a shame most don't know about it and most of the rest sneer at it.