Hacker News Reader

Comment by rwmj | Hacker News Reader

The true answer is buried about half way down the page. The fallocate command / syscall nowadays allows you to remove or insert block-aligned extents at the beginning of a file, assuming you're using a modern filesystem like ext4 or xfs. The only difficulty that remains is working out how many blocks to remove and dealing with the non-aligned part. LWN has a few articles on this topic: https://lwn.net/Articles/589260/ https://lwn.net/Articles/629965/

View on HN

I don't know man, the top answer is clever, but it's also a pretty straight-forward UNIX way of doing it, and pretty easy to understand. Going around messing with trying to convert to filesystem blocks and obscure syscalls that only work on certain file systems seems way scarier in comparison. And you're almost certainly going to end up with extra characters at the beginning, so you having to do some version of the dd thing anyway.

The actual answer, as others have pointed out, is to buy a new drive. If the data has any value, it's worth it.

> The actual answer, as others have pointed out, is to buy a new drive. If the data has any value, it's worth it.

If you read the comments on the StackExchange question, the scenario that prompted it does make sense:

>> I'm migrating our company's database and this is 700GB file with profile data in JSON format, each line one profile. The process that imported the data stopped after running several days after 70% done. Now, to not repeat the full import, I want to cut the first 70% (300 million lines).

>> [One option would be] exporting the data from the old DB again, skipping the first 300 Mio entries and cp it again over to the new and import it there. It roughly takes a day. I just thought there must be a way to do it in place: the data are already there - just truncate the first part of it and continue.

So it's not as if this is the only copy of the data. It would just be nice if there was a way to cut some of it without recopying everything.

In that case, surely it would be possible to point the database importer at a pipe, the other end of which is the output of `tail -n +300000000 db_dump.json`. No need to actually modify the file.

Or, if it's JSON, just replace the first 300M lines with spaces.

You probably don't care if it's 299M rows, so just dd over the start of the file, guessing the number of bytes to write and choosing a low number to be sure. You'll know if you're missing records when you run a count(*) on the database when it's loaded.

Use some hex editor (most support files larger than ram) to check you have the necessary opening square brace at the start of the file, and to remove whatever partial record is there.

Sure, it isn't elegant, but is a solution most people can come up with without needing to spend 20 minutes hunting through man pages.

Yeah, fair enough, that's a reasonable scenario.

I worked at a place where the entire's company business (images) was stored on a single large drive, and the CEO was too cheap to buy a second, so there was no backup. Never underestimate how dumb people can be sometimes.

Was that a special type of drive that cost as much as the company's value? Was there any justification besides the cost of the drive? Even something like 'I live dangerously.'

Inaction, deluding oneself about risks and hubris all have a special value all their own. Very popular these days.

Hey colleague! Of course whatever was used to make those images disappeared some 10 years ago. And the actual mint images are already gone from an earlier disk-crash. THESE images are actually made from systems that have been running at a customer site for years and years.

I've found it curious that so few (any?) file systems allow for non-aligned starting blocks. Ie keeping an offset into the initial block. That would make prepending files rather trivial, as well as trimming the start of files (hello log files).

Of course performance would suffer due to non-aligned access, but just as for regular files one could demand aligned access for optimal speed.

Extent-based filesystems have no problem with that, and performance wouldn't be impacted either. The preserved tail of the first extent would have to copied / CoWed, that's it.

https://man7.org/linux/man-pages/man2/ioctl_ficlonerange.2.h...

From the linked article:

>The immediate user for this operation would appear to be video editing applications, which could use it to quickly and efficiently remove a segment of a video file.

Do any of Linux video editor apps actually employ this feature?

Aren't all video editors "non-linear editing" these days anyway? All they mutate when you're editing is a small amount of metadata called edit decision list. The output file is only created when you explicitly execute a "render" operation, everything you see before that is just parts of the input files mixed as needed.

A segment of a video file isn't going to be block-aligned.

Deleting a non-block-aligned segment of a file means copying the remaining data over it and then truncating the file at the end using the ancient truncate function. It's not obvious how that can be improved.

Without getting into the details of video compression, you can't remove arbitrary parts of a video file without causing problems when the file is played.

Even if the video is uncompressed you will still need to remove entire frames and then - for most formats - reconstruct a file header at the start of the file.

This is a much harder problem than throwing away x% of a file without worrying about the structure of the content.

It's fairly easy to solve that problem at the disk block level. But if you're trying to keep the data structures error-free, you're going to have to do a lot more work.

Reply on news.ycombinator.com