I interned this summer in a genetics lab, and was shocked the extent to which this is the case. Both the input and output files are absurdly huge, so I'm not sure anything else would've been viable, but still, at least as a student it always seemed like there should be "some other way."

After a week or two of getting to know the system, for clarification I asked: "You mean this entire sophisticated system is just some Perl, with a database that keeps track of the flat files?"