They're both generalised solvers, but default Random Forests aren't the most common Forest these days - LightGBM/XGBoost are both using Gradient Boosted Forests by default, which would be a much more interesting comparison to a NN's Gradient Boosting.
What exactly is that?
The 'typical' response is either to make them into numeric variable, so 1-3 for 3 categories, or to make an individual column for each one. The first approach makes sense for ordinals, but not so much for actual categories, and the latter makes it difficult to group categories when a group of two categories together has more predictive capability than any single group. I know that LightGBM did a lot of work in this to optimise testing groups of variables, as testing every possible group in a large set is very intensive.
When I tried Catboost in R, I remember it downloading a large binary to work with, which put me off considerably, and predicting with it was pretty fragile, even for R. I trust Yandex about as much as I'd trust Google, but it seemed 'odd'.
I am pretty sure I've seen this done successfully in kaggle a bunch before, but don't have any sources on hand for evidence that this method is "better". It does however make it much easier to just throw the data into the RF and check the feature importances to see which features are helping the most.
spending too much time on categoricals is a waste of time, there are other things you can improve in your limited time, and even 'doing the right thing' results in poorer performance in hold out.
catboost is great, it ensembles wonderfully with xgboost. if you find it being fragile, you probably have outliers that need droped - tree algs are really just fancy nearest neighbors so an outlier can ruin predictions considerably.
In general, lgbm trains fast and lets you try many things quickly, but almost always under performs catboost and xgboost. catboost performs really well out of the box and you can generally get results quicker than xgboost, but a well tuned xgboost is usually the best. since xgboost and catboost build trees differently and both perform really well, they make great friends in ensembles.
I have done pretty well on kaggle though I haven't invested much time, top 100 in zillow home price prediction
It's more pragmatic than trying to rationalize which framework is "best" for a given dataset, as the results are often counterintuitive.
And that IMO is a reinvention of the "Walled Garden" of academic HPC (ask any grad student begging and pleading for supercomputer time) which has always sucked and its new commercial incarnation is even worse because it's unclear how to get commercial cloud time on government grants.
OTOH it's fine for large shops like OpenAI, DeepMind, AWS AI, FAIR, MS Research etc because they have deep deep pockets. So if you're content with most future groundbreaking research coming from a small tribe of market leaders, well great, but I suspect innovation is already slowing down because of this.
Those hours of hyperparameter search aren't blocking. You can do other things while it's searching, or do the search when not actively using the resources (e.g. overnight).