I totally agree, but want to add two observations:
1. Humanity has already been on the path of exponential growth. Take GDP for example. We measure GDP growth by percentages, and that's exponential. (Of course, real GDP growth has stagnated for a bit, but at least for the past ~3 centuries it has been generally exponential AFAIK). Not saying it can be sustained, just that we've been quite exponential for a while.
2. Not every function is linear. Sometimes the exponentially increased inputs will produce a linear output. I'd argue R&D is kind of like that. When the lower hanging fruits are already taken, you'd need to expend even more effort into achieving the next breakthrough. So despite the "exponential" increase in productivity, the result could feel very linear.
I would also like to add that physical and computational limits make the whole singularity thing literally impossible. 3D space means that even theoretically sound speedups (eg. binary trees) are impossible in scale because you can't assume O(1) lookups - the best you can get is O(n^1/3). Maybe people understand the singularity concept poetically, I don't know.