Not just setting up, but celery has a learning curve which may or may not be worth your time.
Also you have an additional points of complexity (rabbitmq and celery workers) that also have an additional operations maintenance burden.
For a small app that doesn't have to scale, I'd avoid it.
But when building a business focused web app, I use it a lot, especially for large file parsing.