Hacker News Reader

Comment by fragmede | Hacker News Reader

Except that failure modes of ftp (technically not ftp-the-protocol itself, but what the server chooses to do with the file) is not well defined. What happens if the connection dies half way through? Is the partial file processed? None of it? Does the file get moved after the upload is done or after it has been fully processed? Does every single last company in this space's ftp site behave the same in the face of errors? Also is this literally ftp and not anything more recent that includes encryption, but if it includes encryption, what ciphers are supported? Nevermind the files may be in ebdic or something else wonderfully obscure...

(I also work in payments. Bank's SFTP sites have under-defined failure modes.)

View on HN

As do I (bank file transmissions representing!) To compensate this partial file fiasco, we tend to rely on the .done file methodology (i.e. we won't pick your file from your server until your script writes out a dummy file we can locate). Or we'll allow you to just push the file to us. Our system will notice partial file send of course, where the transmission stopped. But we can't determine if the file was partial to begin with. So we rely on balance reports to come via alternate FTP or email transmissions.

Don't get me started on sending ASCII as binary to the mainframe to compensate for the EBCDIC formatting. Or the lack of carriage return and line feed characters that cause so many fun issues.

All of that to say that none of it's pretty, but all of it works. The balancing is key as are the extra staff needed to verify them against each other.

Can this be solved by sending two files, one with the data and one with a checksum of the data?

That's a reasonable idea! There are a wide variety of ways to solve the problem, using ftp uploads as the primitive, but ultimately that's... kind of the problem. Everyone's solution is different from everyone else's, but those different solutions have different ramifications, so when they fail, they have to handled very differently. That is to say, OOP, classes and inheritance, only gets you so far.

(An issue w/ whole file checksums is that there are cases where partial file processing is desirable, but that's not to say there's not use of checksums.)

That's what we do. Have a summary file and a detail file and they need to match.

It's also bad when your trading partner's sender process crashes and they build up a backlog of a million messages, and then once they fix it, they want to dump their whole backlog on you at once.

Or their process gets stuck in an infinite loop and resends you the same message millions of times, etc.

Reply on news.ycombinator.com