Lexer is just a parser for a regular language. They output a linear stream of tokens, conceptually similar to a stream of characters, only smaller in size and easier to create a grammar for since superfluous elements like spaces are discarded. You could totally parse at the character level if you want to by treating each character as a token, it's just that it makes things needlessly hard.

The general rule seems to be: lexer is for parsing linear structures like numbers and symbols while parsers are for recursive nested structures like lists and objects.