In my non-chemist, non-doctor opinion the circumstantial evidence points to brake dust being way less bad than tire dust.
Being inert in the presence of most common substances tends to correlate well with low toxicity. It's the stuff that chemically reacts to all sorts of things that tends to be the most poisonous.
Tires break down over time under "normal earthly conditions" which includes a lot of the same temperatures and molecules that are found inside the human body. To me this is a massive red flag for toxicity.
High temperature tends to make things react more readily. Most of the stuff in brakes has to be really stable at high temperatures and being stable at high temperatures tends to correlate well with being chemically inert.
The most reactive stuff in brakes are the long complex molecules used to bind it all together. These sorts of molecules are similiar to the molecules used in tires and all sorts of other plastic like things. So all else being equal tires probably have more of the chemically interesting things that cause problems in the human body.
Burning brakes smells way better than burning tires which seems to confirm this (remember, your nose has millions of years of incremental improvement when it comes to indicating what is and isn't suitable to ingest).
And before anyone thinks they're gonna get some easy internet points by saying "but asbestos", the amount of asbestos in brake pads is nonzero (to account for the sketchy unbranded stuff that may or may not use it) but still tiny enough that people working in proximity to it have mostly stopped dropping dead at 60 thanks to lung cancer so I think that however much asbestos is in brake pads it's low enough to not be an issue from a public health POV.
TL;DR I'll take my chances with brakes over tires.