No, you can E2EE on all clients. But Secret Chats are always a 1:1 session between devices. You can't start a Secret Chat on one device and continue on another.

This might seem like an inconvenience but it avoids the pitfall leading to the recent Matrix exploits...