How do I delete bytes from the beginning of a file? (2010)

Vim's not line oriented... at least as far I know. ed/ex, now those are line oriented.

But I wouldn't call vim line oriented. Unless you also want to call Notepad line oriented, since you can use Shift + End to do operations on an entire line :D

Right—the difference is that, with a file object, the extents are all assumed to be full to capacity except for the very last one, with the information on the virtual size within the last extent being held in the file's inode.

Whereas, with these vector-of-buffers objects, either each extent would need to reserve two bytes at the beginning to represent the virtual size of that extent, or the vector would be a vector of disk slices at byte granularity, rather than a vector of plain disk offsets at sector granularity.

I can see the arguments for both.

The former is extremely efficient, and would probably be the go-to for databases that want to be performant without needing to dedicate a raw block device to them.

The latter approach, though, you can implement entirely "on top of" the filesystem (in the same way that the ext3 journal, or APFS containers, are implemented "on top of" their filesystems.) You could have a regular file representing the storage reservation (in full extents) for the vector-object; and then the vector object could be a tree (or something else efficient to rewrite) of byte-granular slices of offsets into that backing storage-file, with logic to ensure that such a reserved slice never straddles two extents, and a cache in the entry for a pointer to the actual extent. These would be perfect for, say, generating archive files.

In fact, since these are such different use-cases, there's no reason to not just give users both.

Of course you can do it yourself; but the point would be to have a standard API (like files are a standard API) for interacting with such on-disk vector-of-extents objects, such that one program could write them and another program, written by a different author, could read them back.

The simplest obvious use-case for pure vector objects: storing Unix text-based file formats as vectors of line-length extents, rather than a single byte-stream. wc(1), head(1), tail(1), etc. would now be O(log n) operations. Any parsing operation that can proceed separately for each line (like the context-free phases of syntax highlighting... or like grep) could be parallelized. The .mbox file format would be relevant again. This would essentially do for Unix what record-orientation does for z/OS—while keeping things "Unix-y."

Another powerful use-case: concatenation. The direct ability to manipulate extents means the ability to compose sequences of extents together into new files, without having to copy the underlying extents. So you can download some large document in parts (either because it's explicitly chunked, ala partial RARs; or because you used range requests to retrieve the file one chunk at a time, perhaps in parallel, as in HTTP or BitTorrent) and then just concatenate the chunks back together in O(log n) time. No need to pre-allocate a file, target your chunk-writes, and keep an external index of which chunk-jobs have succeeded. Just keep a downloads directory and concatenate when you're done.

It's funny; modern filesystems have the theoretical capacity for many operations like O(log n) concatenation, but they only bother to expose one or two (hard-link, and sometimes copy-on-write clone/snapshot). Filesystem authors are seemingly unimaginative and conservative where exposing new APIs is concerned; I wouldn't expect them to ever directly expose the ability to O(log n) concatenate files. But if they exposed a vector-object, it would give you these features for free!

But maybe that doesn't seem useful enough. Let's add one more feature: give the ability to store a "table of contents" with this vector-object, i.e. a labelled interval tree where the extents in the vector are the leaves.

Such a filesystem object would entirely obviate specialized tooling for container/record file formats; rather than mkvtoolnix(1), or avro-tools, or LevelDB, you'd just need a program that took the raw byte stream of one of these container files as input, and spat out a labelled-interval-tree-extent-vector file-system-object representation. You could cd into it, put things in, take things out... it'd seem a lot like a directory. But it'd still fundamentally be a file, in the sense that it'd still fundamentally be a bijection with a seekable byte stream. (Unlike actual directories, where dirents have no standardized collation order; where there's no obvious semantic answer for what part of the "content of a dirent", if any, goes into a treewise cryptographic checksum; etc.)

Certainly, though, if you had this type of object in your filesystem, you could implement your filesystem's directories in terms of it.

I believe record-oriented data sets only give you fixed but not arbitrarily/variably-sized extents. I.e., a given file (data set) is of one record type (sort of like an RDBMS table is of one record type), and the record type then determines the extent size for the whole file.

Whereas, what I'm talking about, is more like what you get by having a sorted-set data structure in Redis, and putting arbitrary strings in it. Or storing data in LMDB with BASIC-like numeric labels as keys. Rather than a tabular representation, your data store is a heap, with a tabular index set aside to point into it.

(Of course, you can emulate arbitrary-length records over fixed-length records, as Postgres does with its TOAST tables, or—in the small—as UTF-8 does with runes. But in the record-orientation implementation, that’s something that would be exposed to the developer, requiring them to solve the problem themselves or through a library; rather than the file system just showing you arbitrary-length records from the start. And, if you can’t rely on other users doing that work, you can’t really use it as an interchange format.)

Has anyone written a high-level introduction to mainframe tech that's accessible to people outside the mainframe culture? I've always been curious about it, but the IBM manuals I've come across have been somewhat impenetrable.

The Commodore 64 supported (or maybe it was the 1541 drive that implemented it) what was called "relative files", which were composed of fixed-size records that were accessed by index. I always thought they sounded cool, but never had a real need for them in my toy projects on that platform.

Maybe I didn’t make that clear, but that’s what I was talking about. That’s why I used the term “vector” instead of “stack.” A vector ADT usually allows insertion/deletion/replacement from the middle.

Actually, most file systems let you randomly read and write to files. Usually, they implement this by reading and writing blocks (of 256 bytes or a multiple). One could create file systems with more advanced operations, but most likely, these will be based on the more primitive operations. For example, to add the function to remove 10 bytes from the start of the file, one could read the whole file, block-by-block, and move the data 10 bytes. For a large log file, this will take a long time. Another solution is to have a begin of file offset for all files, this means that some additional information need to be stored with every the file and that this information needs to be accessed with all file operations, resulting in some performance loss for all file operations. Existing programs that access files as blocks of a certain size, might see significant performance penalties when the offset is not a multiple of the block size, because reading the data from one block, needs to read two actual blocks from the underlying medium.

If you often want to strip a logfile from old data, a solution might be to create new logfiles at a regular intervals and delete old logfiles. Or maybe even better, use a database to store the log messages. Gives you query functionality and allows you to perform more advanced operations, as removing certain types of log messages.

It will remove if you use FALLOC_FL_COLLAPSE_RANGE

The "native" sector size of an SSD is the erasure block size, typically 64KiB.

current SSDs are page/block oriented, but that's likely to change. (See intel's relatively recent Optane).

Yes it does? The details are file system specific and if they support mmap then it's also likely there's a page alignment requirement but that's not atypical.

Not only does it do so, but dd directly supports sparse files, using the "sparse" conv symbol.

It's unnecessary to create a filesystem-level support I think, since you could make a circular buffer holding a video stream in a file with some constant size that is approximately n hours of video. With some clever tricks you could even keep the format compatible with existing video container formats.

Journaling filesystems do that internally.

Curiously enough. PalmOS's file streams supported this very feature: delete as i read. As you read bytes from the start of the file, they'd be deleted. You were allowed to have a concurrent writer appending to the end.

Linux and fallocate() allows precisely this - we use this to "log locally" for applications, and a "tail this asap and publish to kafka then delete locally" helper.

See ( https://gist.github.com/minaguib/1cbe29922b06d50755a2f580b8c... ) for some test notes I took a couple of years ago.