Hacker News Reader

Comment by petercooper | Hacker News Reader

This is relevant but forgetting torrenting for just a moment.. I believe Dropbox uses hashing to identify files in their system, since sometimes you can copy a large file into your Dropbox and it syncs immediately (without an upload).

Given this, rather than using torrents, could there theoretically be a way to tell your Dropbox account you "have" a certain file that, indeed, you do not, merely by using the hash? For example, if I know the hash for the latest episode of a TV show is "ab12de" (gross simplification!) and I can make Dropbox think I "have" that file, if someone else already uploaded that file to their Dropbox, I could grab it too?

View on HN

This has been done and the exploit has been (mostly?) closed. Google "dropship" for all the details.

How does their dedupe method work now? Shouldn't it still be possible under the same principle?

Client: I have the file with hash 0x47fed9.

Server: Okay, tell me what byte 1257 (randomly chosen) of that file is and I'll mark you as having the file.

Client: Ummm ...

The way you said it is x100 times better than just explaining it.

Thank you

Do they ask for exactly one random byte for every 4mb hashed block?

For a <4mb file, if you just guessed a random byte and try 256 times, wouldn't you probably get it right at least once?

Still, they could easily change the scheme to ask for more bytes to reduce the chance of a correct guess, plus it would be hopeless for large files.

They could give you a sufficiently long random salt and ask you to calculate the hash of the salt and block contents. This scheme seems to be secure.

So a client could theoretically talk to another client that knows what byte 1257 is and the file would appear to answer what the question implied.

I think you missed the part where byte 1257 was "randomly chosen"

It depends. If I already have the full file, it's much easier (and efficient) to tell you the value of a random byte and let DB give you the file than having to transfer it all.

Of course, but then you still need infrastructure to figure out who else has the file, how to keep things private, etc. You can't just distribute a hash and be done with it.

If you're dealing on a small scale (e.g. "Hey, icebraining, let me grab that file from you!"), it's going to be easier to just send a share request from your Dropbox account to mine, which has the same end result (flipping an ownership bit in Dropbox's system).

That doesn't really scale, no, but sharing hashes doesn't really either, unless you want to spend a significant amount of time building infrastructure to keep track of who has what files and automate the management of sharing hashes.

I didn't miss that part, I was simply saying that a client that had the file could tell what byte 1257 was to the other client (or some other random byte, as requested by dropbox).

If there is cooperation with another client who has the file you could just get the file from them in the first place.

Perhaps the bandwidth between (dropbox + all your syncing devices) is greater than what the conspiring client can provide, so such a scheme may still be beneficial.

I think they could just share the folder with you on Dropbox, allowing you to download it from there.

You don't need to "get" the file from anyone, you just need convince Dropbox to mark your account as having that file.

Maybe I'm missing something. What possible purpose (other than getting the file) could there be to convincing Dropbox that you have a file?

By "get" I just meant download directly, which could be slow and eat a lot of bandwidth. And then if you actually want that file in your Dropbox, you'd eat even more bandwidth doing the upload again. But syncing once from Dropbox is faster than downloading from someone else and then uploading to Dropbox and then waiting for your other computers to sync. And since you don't have to get the file from the original sharer, they don't have to have enough bandwidth to go around, just let Dropbox handle the copying.

Other than? "Having files" is the entire point of Dropbox.

>> You don't need to "get" the file from anyone, you just need convince Dropbox to mark your account as having that file.

That post made it sound like there was a purpose to having the file in dropbox other than getting it.

So I forward the request to a peer who has the file, he tells me the answer, and bam. Doesn't the BT protocol allow the download of small segments of a file?

Yes and it is a win win result: you have proved that you have access to the file, so dropbox has only optimized bandwidth.

How long does it wait for an answer? Basic maths will allow you to work out what Chunk is required from the torrent swarm to get the answer pretty quickly.

Clever...

Google "dropship dropbox". Googling "dropship" wont get you anywhere.

Im pretty sure that the way this works is by their own server downloading the file into a dropbox folder on the server. Then they use a copy-reference (https://www.dropbox.com/developers/reference/api#copy_ref) and pass that to the user who will then receive the file. So, in essence is the same as what you said, only using the reference instead of the hash. This reference stuff must be the way it works because you cannot upload a file greater than 150mb through the dropbox api. Lastly, the way that dropbox prevents abuse of the copy-ref api is by bandwidth limiting the reference.

Google App Engine's SDK used to (still does?) use a very similar scheme.

Reply on news.ycombinator.com