Complex proc macros absolutely do slow builds down. In many cases, a proc macro only need to be a stub that can delegate to a declarative macro.
You may not need to use syn/quote, but if you are doing any sort of processing/parsing of Rust code you pretty much need to.
FWIW, I really hope that the Rust project focused on finer-grained token matching in declarative macros so we can migrate most proc_macro code away. The macro system is powerful, but nowhere near where it needs to be.