Color just means putting red, green and red filters in front of the pixels in some pattern and so it is definitely not a principle problem but you will lose a large part of the light spectrum for every pixel and therefore harvest a lot less energy.
You could have separate smaller color sensing pixels, using only a fraction of the surface area.
ETA: There are also transparent solar panels which only absorb UV. If that were tuned for visible colors, you could have three transparent layers over an ordinary panel, and maybe not lose anything.