Another variant is four words: "we had an agreement." This emphasises that you had already agreed to something.

I agree with everything else, it's very unlikely to work out.