First, I'd say I don't think you have anything to worry about.
Second, don't go replacing MySQL because you're worried about scale. I have yet to find a database that really easily scales better. You can trade a whole lot of development time to use something like Cassandra or Riak, (which are both great for particular applications). Then you spend a huge, huge amount of time building queries that were easy in Rails and you actually didn't solve any end-user facing problems. You just made it where only one of your developers actually can create queries and nobody can maintain the code.
Third, don't go replacing Rails because you're worried about scale. It's not the fastest thing but it's just not worth losing sleep over.
Just add the appropriate technology where you need it - I probably wouldn't build a recommendation engine in Ruby; perhaps that is the only piece that you build using something else.
Here's what we often do @ Inaka - web and admin in Rails, business "logic" and stuff that needs to scale (either because we have tons of socket connections or because we have a lot of data to crunch) in Erlang. (Insert Java or Python or whatever you need for the backend piece in place of Erlang.)
Then build an internal HTTP API that Rails can talk to. Abstract that in a model class. Have Rails call into it as needed. RecommendationEngine.recommend(...) returns JSON with the magical results for your Rails app to render appropriately.
Typically we give our Rails users an "api key" (that they don't know about) that is used to authenticate calls to the backend service, so we don't even have to share user authentication schemes between the systems. Then you can use devise or whatever you want for Rails but don't have to re-implement the same password hashing algorithms - authenticating those users is just a quick lookup in the user table.
Sometimes that service API may even be part of a publicly available HTTP API. For instance, imagine the backend piece exposes part of its API to mobile devices. Then, some of the methods may be authenticated and some are open to the world.
PHP doesn't scale. Java doesn't scale. Python doesn't scale.
If you wait for something that "scales", you'll never ship anything.
Scaling isn't done automatically, it's something that's applied to a problem. If MySQL doesn't scale, and it has proven itself to be very capable in a wide variety of circumstances, then what does?
The more you read about scaling, the more you realize there's no magic bullet, no magically scalable language or platform or framework. It's all about careful investigation of the nature of the problems, the bottlenecks, and developing solutions to address those.
I shall get back here to post a link to our product in about a week when we launch. Would very much appreciate comments then!
This performance problem could be because of missing indexes, grossly inefficient queries, or a whole host of other elementary problems that can be fixed with a keystroke.
If you find that MySQL is a bottleneck in particular, Ruby (and by extension Rails) has a wide range of database adapters and database ORM's available. Active Record itself has numerous adapters, which mean changing databases within the supported set shouldn't be too difficult.
Alternatively, you could use a different ORM such as DataMapper or one specific to the database you desire. Rails now let's you choose which ORM you wish to use, although if you're just getting started it might prove an additional learning curve as most of the documentation uses Active Record.