I first encountered it in the 1981 James Clavell novel Noble House. The character using it was a well-educated Hong Kong gangster, or something similar to that.

(The plot of the book revolves around massive favors that a certain character is obligated to fulfill. At one point it is argued that "the ask" for one of them, while greatly annoying, could instead have been worse yet.)