Indeed, cartpole should be that easy. The observation vector is dimension 4, and the action vector is dimension 1. Gym has a range of difficulties, of which this is the lowest. It's good for debugging because you can verify the math in your head.

The linked code is quite fragile, though. It chooses random models (4-vectors) with all positive signs between 0 and 1. It so happens that there's a valid solution in this range, but a good RL agent should work equally well with flipped signs of observation and action vectors, and it should be able to solve it (possibly with more iterations) with any affine transformation of the two vector spaces.