Hacker News Reader

Comment by newnewpdro | Hacker News Reader

The filesystem just needs a concept of a file start offset.

The initial value is 0, if you wanted to support truncating at the head, you just add to this offset the amount truncated.

In a sector-based filesystem, whenever this offset goes far enough into the file that crosses sector boundaries, you reclaim those sectors as free space. When it lands within a sector somewhere, that sector is pinned and you waste some space.

The problem is more that the userspace APIs don't expose well-supported mechanisms for doing this. Implementing it at the filesystem level is trivial.

View on HN

I thought his point in the blog was that it was indeed trivial to implement but when you think about all the consequences it becomes trickier to do it well.

He makes that point at the end.

I can think of a cool case for this.

What if you have an append-only system which is making remote backups. One service is writing the activity log append only, a second service is reading that, check-pointing, and then truncating the head of the file when the check-point has been committed. No need to do the tricky file-swap trickery.

Journaling filesystems do that internally.

Curiously enough. PalmOS's file streams supported this very feature: delete as i read. As you read bytes from the start of the file, they'd be deleted. You were allowed to have a concurrent writer appending to the end.

As you read bytes from the start of the file, they'd be deleted. You were allowed to have a concurrent writer appending to the end.

Also known as a named pipe.

Kind of, a pipe has a limit of 16 pages which is usually 64kb.

But with no limit on buffer size. Pipes have a limit after which writer blocks. Also pipes aren't persistent across reboots. These are.

Linux and fallocate() allows precisely this - we use this to "log locally" for applications, and a "tail this asap and publish to kafka then delete locally" helper.

See ( https://gist.github.com/minaguib/1cbe29922b06d50755a2f580b8c... ) for some test notes I took a couple of years ago.

fallocate() is not supported on network file systems, is it?

This is discussed in TFA and a number of problems with this "just do" solution are pointed out.

Reply on news.ycombinator.com