The introduction was not very satisfactory about who they are so I checked Wikipedia in French. The hypothese is they are descendants from Goths, cornered in South-West at some point of history by Franks.
The introduction doesn't explain this because, as the article goes on to explain, nobody knows what originally distinguished this group. That's the whole thing about the cagots. There are a number of hypotheses including the one you mentioned about the Goths, which are all described if you read further.