Today I learned that $HOME/Library/Preferences/com.apple.LaunchServices.QuarantineEventsV2 is a sqlite database. The LSQuarantineEvent table contains every file you ever downloaded, the source URL is in the LSQuarantineDataURLString column.

sqlite3 $HOME/Library/Preferences/com.apple.LaunchServices.QuarantineEventsV2 'select LSQuarantineDataURLString from LSQuarantineEvent'

sqlite3 $HOME/Library/Preferences/com.apple.LaunchServices.QuarantineEventsV2 'delete from LSQuarantineEvent'