Hacker News Reader

Comment by OskarS | Hacker News Reader

I don't know man, the top answer is clever, but it's also a pretty straight-forward UNIX way of doing it, and pretty easy to understand. Going around messing with trying to convert to filesystem blocks and obscure syscalls that only work on certain file systems seems way scarier in comparison. And you're almost certainly going to end up with extra characters at the beginning, so you having to do some version of the dd thing anyway.

The actual answer, as others have pointed out, is to buy a new drive. If the data has any value, it's worth it.

View on HN

> The actual answer, as others have pointed out, is to buy a new drive. If the data has any value, it's worth it.

If you read the comments on the StackExchange question, the scenario that prompted it does make sense:

>> I'm migrating our company's database and this is 700GB file with profile data in JSON format, each line one profile. The process that imported the data stopped after running several days after 70% done. Now, to not repeat the full import, I want to cut the first 70% (300 million lines).

>> [One option would be] exporting the data from the old DB again, skipping the first 300 Mio entries and cp it again over to the new and import it there. It roughly takes a day. I just thought there must be a way to do it in place: the data are already there - just truncate the first part of it and continue.

So it's not as if this is the only copy of the data. It would just be nice if there was a way to cut some of it without recopying everything.

In that case, surely it would be possible to point the database importer at a pipe, the other end of which is the output of `tail -n +300000000 db_dump.json`. No need to actually modify the file.

Or, if it's JSON, just replace the first 300M lines with spaces.

You probably don't care if it's 299M rows, so just dd over the start of the file, guessing the number of bytes to write and choosing a low number to be sure. You'll know if you're missing records when you run a count(*) on the database when it's loaded.

Use some hex editor (most support files larger than ram) to check you have the necessary opening square brace at the start of the file, and to remove whatever partial record is there.

Sure, it isn't elegant, but is a solution most people can come up with without needing to spend 20 minutes hunting through man pages.

Yeah, fair enough, that's a reasonable scenario.

I worked at a place where the entire's company business (images) was stored on a single large drive, and the CEO was too cheap to buy a second, so there was no backup. Never underestimate how dumb people can be sometimes.

Was that a special type of drive that cost as much as the company's value? Was there any justification besides the cost of the drive? Even something like 'I live dangerously.'

Inaction, deluding oneself about risks and hubris all have a special value all their own. Very popular these days.

Hey colleague! Of course whatever was used to make those images disappeared some 10 years ago. And the actual mint images are already gone from an earlier disk-crash. THESE images are actually made from systems that have been running at a customer site for years and years.

Reply on news.ycombinator.com