Return-to-libc (a.k.a. ret2libc) is usually used to bypass NX-bit / DEP, but NX was disabled for level04 - the solution does not require return-to-libc.
I think I'll try again tomorrow morning and see what I missed.
ASLR actually makes ret2libc much more difficult to pull off (esp. on 64-bit platforms), because the location of libc.so itself is randomized.
Also, there's another useful place you can return to that isn't in libc.