You don't need a full scan or a giant CSV file. You can do it with a single small file in a single directory. The file contains the state of a pseudo random number generator. On each request, generate a number, save the new PRNG state into a new temp file in a subdirectory, and rename the temp file over top of the original file. If renaming fails, try the whole process a few more times. This allows multiple processes to serve requests concurrently.

Exercise for the reader: avoid re-reading the PRNG state file on retries.