I retired in my mid-thirties and almost everyone I told before hand had the same or similar pejorative response as yours... "Oh I could never do that" or "I would be bored" or even "Why would you want to do that?". I ended up stopping to tell people.
There is so much you can do when you've disconnected your employment from your living expenses. Not _having_ to work is very different than simply not working. You can work on what ever you would like. You can ask to work 2 days a week for 20% pay. You can work on mastering something completely different like woodworking, cooking, and instrument, or even something around software development. But to just associating FIRE with a monk-like or sedentary lifestyle is an overly broad and often inaccurate representation of the community.