It's particularly difficult with most proteins and other biomolecules. [0] But even for small molecular compounds, it isn't always easy. I'm not sure about growing high quality crystals of metal-organic frameworks, but they could easily be limited by defects. Some compounds might be too waxy if there are long aliphatic chains involved.
Even if you do get decent-looking crystals, they might be twinned. If you are working with air or water sensitive compounds, they need to be handled with extreme care and might degrade before the data collection is complete. Some might also have temperature sensitivity, such that they need to be kept at -20 C to prevent melting or another phase transition.
Overall it can be a real make-or-break issue for someone engaged in chemical research.
[0] Check out this PDF of a PowerPoint on the topic: https://hamptonresearch.com/uploads/documents/ramc/RAMC2011_...