Yes, absolutely. The move has to be done gradually to allow farming to adapt. An overnight cutoff would surely cause yields to drop.
What isn't clear to me is how much warning was given for the ban. Was it announced ahead of time?
In the northern parts of the country, where the majority of rice farms are, a considerable percentage of people have kidney diseases that attribute to the use of chemical fertilizer.
There is a surplus of rice production in the country, so I don't think the reduced crop will make a big difference.
However, the state of tea farming is not the same. Price of Sri Lankan tea (Ceylon tea as it's commonly called) at auctions was slowly going down for the past few years, and tea estates expect the government to make an exception for them.
Since this ban was announced, the finance minister was changed to the brother of the current president (another can of worms), who might make necessary changes, exceptions, or probably just delay it to make sure the tea industry isn't disrupted.
https://www.france24.com/en/live-news/20210901-sri-lanka-org...
Rajapaksa came to power in 2019 promising subsidised foreign fertiliser but did a U-turn arguing that agro chemicals were poisoning people.