Yes of course there is a dollar amount where it stops becoming worth it, but the primary rule is simply whether the amount is sufficiently negligible. It is not whether the amortized return on investment beats the specific dollar amount.
As a thought experiment, would you pay $500,000 a month if the payoff was $5B?