What's the difference between a startup and a small business?

What makes a startup a startup? Does it have to be technology-related? If I start a small business selling widgets or a service, are the same elements not involved: some capital injection (from savings or family loans), one/two owners acting as CEO, CFO, CIO, Sales, Marketing, HR, Ops, etc., juggling many things and struggling with the same questions?

I am truly curious - what constitutes a startup and how is this different from the thousands of non-tech businesses that are started every day?