Most people do not use secure passphrase to lock their phones, but use a PIN. Even an 8 digit PIN is only 10^8 guesses, which any modern desktop can crack. To solve this, Android/iOS make an extremely long salt, so that + your PIN is your passphrase. In addition, there is logic to deter brute force attacks (e.g. after 10 guesses wipe my device)
The attacks have to overcome both of those, but if they do, cracking the original secret (your PIN) is easy.
The other method is attacking a "live" phone/laptop, as it has the unprotected key, and one just needs to exfiltrate it.
If you are truly concerned, DO NOT USE a trivial crackable PIN! Use a long password that is hard to crack by itself. Additionally, turn off your device if you think you are in a position where you have to surrender your electronics! That way the unprotected key is gone.
(As a side rant, I wish Android and iOS would allow you to have a "start up" password and a screen unlock password. You used to be able to do that on Android, but it was through root and not officially supported. But it would be nice because then you could have an easy to unlock screen password and a really difficult encryption password).