There are many tradeoffs. The good news is that hardly any projects and startups will hit those. Or, when they do, they have the resources and funds to transition.

Anyway:

Ruby is slow. Terribly so. Rails even more. The optimisation is on speed of development and that comes (a.o.) at the cost of runtime speed. Memory usage is terrible too. Easily solvable by bigger or more VMS though.

Ruby, with its runtime, is complex and hard to build and deploy and host. Heroku et al take that away from you, but "getting it running on your laptop" alone is a sysyphian task. I've worked at and managed teams where this remained a timesink: where devs spend significant time "Debugging their environments".

Rails is simple MVC with loads and loads of magic. Opinionated patterns such as their own take on ActiveRecord, MVC make this worse. If you dislike these patterns or architectural choices: you'll have a bad time. If you want, or need to implement other patterns (such as, but not limited to, hexagonal architecture, event-sourcing, adapters, dependency injection etc) you'll have a hard time and Rails will work against you. Very hard.

Related to this: It is hard (but not impossible!) to keep an ever growing Rails app clean, agile, and stable. Rails works against you in this: it's often easier and quicker to Do The Bad Thing that it is to Implement The Proper Abstraction Or Pattern. It optimizes for speed of development - perfect for MVPs or PoCs, at the cost of maintainability.

Ruby is weakly typed. If you want, or need the protection or guidance of a compiler in typing, this, Ruby won't help you. Additional tools like (complex) linters, static code analysis help and are required to keep a long running project workable. Extensive testing of everything is The Solution Of Ruby to all this: need to ensure your method works when it gets a null (nil) passed: write tests: no compiler will help you.

Ruby is dynamic to the extreme. Many people will feel very uncomfortable with this (only part rightly so, IMO). But a practical downside is that no IDE or vim-setup can help you like you may be used from Java, .Net, Rust or C(++). Refactoring is basically "rerunning the tests over and over. And over. And again".

Edit: and some smaller, but, depending on case, important too:

Rails' fetish with ActiveRecord makes nearly all Rails apps extremely tightly coupled to the database and the data-model. To inverse this: you're data-model is bounded by what Rails needs and wants (in practice: in theory this can be worked around). If you consider that all long-lived projects carry "data" along the journey, but that codebase is iterated constantly, having such tight coupling and rigid requirements on your data-model is a big risk.

Rails' is ever-growing. It's "batteries included" idea is great for starting. But that makes it ever growing in size and complexity.

Rails' architectural style or modules and includes make the common way of writing things hard to test, hard to maintain and hard to debug. For example ActiveRecord::Base is a terrible violator of the Single Responsibility Principle: your models connect to the database, manage storage, manage state-machines, can render themselves, can serialize themselves, manage their validations, manage and coordinate listeners and callbacks, manage identification, have a giant API for fetching and querying in the database, manage their dependencies on other models, manage other models, and so on, and so on. And that when "testing" is the most important tool to keep a Rails app solid and stable.