It's documented over three consecutive chapters:
https://www.nongnu.org/txr/txr-manpage.html#N-5C9CED75 (Structural Pattern Matching)
https://www.nongnu.org/txr/txr-manpage.html#N-B5350C53 (Pattern-Matching Notation)
https://www.nongnu.org/txr/txr-manpage.html#N-CD6130ED (Pattern-Matching Macros)
It's implemented in one file: https://www.kylheku.com/cgit/txr/tree/stdlib/match.tl
Starting with the second definition of the function non-triv-pat-p (the real definition which replaces the temporary stub one), the module starts to eat its own dogfood by using its own pattern matching.
That function, for instance, decides whether a pattern is "trivial" or not, using pattern matching itself on the pattern matching notation.
The string quasiliteral pattern's expansion relies heavily on pattern matching, also.