It might be easier to grasp if you use 1x. E.g. "it's 1x as fast" means it's exactly the same. You're applying the multiplier directly to get the new speed. 0.5x as fast means it's half as fast.
"It's 1x faster" means you add the result of the multiplication to the initial value. So it's twice as fast. 0.5x faster still means it's faster, you add 50% of the initial speed. This way you cannot really express that something got slower, except by using negative values, which might be rather confusing.
I think this might also work in most other languages originating in Europe.
To my ears, "3 times faster" has the same meaning as "3 times as fast".
Some people want to use language that is logical and consistent. For them, "3.5x faster" means the same as "350% faster" and "4.5x as fast". Others believe in convenience and redundancy. They think that "4.5x faster" means the same "4.5x as fast", because the numbers are the same. Many of them don't like expressions such as "X% faster" for X >= 100, because the numbers tend to be misleading.
I used to be in the former camp when I was young. Today I'm middle-aged and lazy. When I see "3.5x" in the text, I assume that it means "3.5x". I don't want to read the text carefully to determine that you actually meant "4.5x" when you wrote "3.5x". I interpret the "faster" part as redundancy. It tells me that we are talking about speed and that the thing we are talking about is faster than the baseline.