Once upon a time, the digit for 0 was a dot, the digit for 1 was (close to) ι, the digit for 2 was (close to) μ, the digit for 3 was close to w with a leading descender like μ has. The digit for 4 might have been closer to a + sign that eventually morphed to look more like a backwards 2. It was the number of vertical lines which encoded 1, 2, or 3; with 4 and 5 and 6 there was more need to come up with more-abstract symbols.
In Arabic these first 4 digits have actually survived relatively unmolested, ٠ - ١ - ٢ - ٣, while the backwards 2 for 4 started hooking down to become a ٤ and some other stuff happened with the next 5 digits.
In English the digits for 2, 3, and 4 all rotated about 90 degrees counterclockwise while the tail disappeared for the 3 and the 4 didn't close up on the top until relatively late.