Unspecified, really? cppreference's [C documentation][1] says that it returns zero. The [OpenGroup][2] documentation doesn't specify a return value when the conversion can't be performed. This recent [draft][3] of the ISO standard for C says that if the value cannot be represented (does that mean over/underflow, bad parse, both, neither?), then it's undefined behavior.
So three references give three different answers.
You could always use sscanf instead, which tells you how many values were scanned (e.g. zero or one).
[1]: https://en.cppreference.com/w/c/string/byte/atoi.html
[2]: https://pubs.opengroup.org/onlinepubs/9799919799/functions/a...
[3]: https://www.open-std.org/jtc1/sc22/wg14/www/docs/n2310.pdf