The mistake isn't investing time into Flutter; it's being "the Flutter guy", only knowing Flutter and only searching for Flutter jobs.
Having Flutter in the toolbox is quite a good thing, unless there is nothing else in there. Then you are in trouble.