Assuming you did this in a day or less I think your prospective employer kind of missed the point of the exercise. Turning out a (fairly well put-together) working prototype in this timeframe shows that you can code, your first-pass level of quality, and that you know how to prioritize.

The output of this project should have been a starting point for further discussion. Instead what you describe tells me the company you interviewed with is probably a "one-pass" (it's expected to be perfect the first time) shop and likely have loads of technical debt. Competent software shops understand the long term nature of development and the tradeoffs involved. They missed an important opportunity to learn about your true depth and ability to work with others by not having you do a paired refactoring of some aspect they didn't like.

Just as people are often bad at being the interviewees people can be equally as bad (or worse) interviewers. There are loads of jobs out there for you. Don't let things like this discourage you.