Querying by rsid is clearly a bad idea. You want to partition by chromosome (and sample-id in this case) and sort by position. When looking for a given snp, the parquet reader will go through the metadata to only read the data page that contains the given position. Unless your pages are huge, read time will be super small (and cost-efficient since you don't fetch too much data). Since the data is static I would want to try storing all the sample data and metadata in arrays. (For non-static data you can't do that because you won't be able to edit the arrays later - you can only add new rows to the parquets.) I am not really sure I understand what the author is doing, sounds like he wanted to sort by position but he failed to do so and decided to bin instead? I agree that Awk is very useful in this kind of problems.