It's usually clear from context.

That sign in particular says "Martial Arts Hall" ("budokan" / 武道館) from right to left.

Imagine an alternate universe where English could also be read left-to-right or right-to-left. If you were to see a sign saying "Hall Arts Martial", you'd immediately know the right way to read it.