You're not alone. A lot of native English speakers get it wrong too. And the other way round as well...

"Will you borrow me ten quid?"

"No. But I'll lend you it"