Japanese e would only ever be [ɛ]. The constructs to produce an [e] sound would be transliterated as "ei" (the quoted are just letters, not meant to be IPA).

So, while I disagree that they sound "exactly the same", I do agree that in most cases the differences between [e] and [ɛ] wouldn't be enough to cause confusion between speakers of either language.