I have a massive soft spot for Harvest Moon because I wrote my first code ever (in Python) to figure out what was the best crop to plant in summer - the answer was turnips, and 17 years later, I still remember that.
The crucial difference for me between AC and SV is that the former all but forces you to take a relaxed approach. SV felt more like a typical game where I tried to do everything the 'best' way, and optimize and 'level up' things. Which is a ton of fun, but also a bit stressful.
In AC, on the other hand, everything is real-time. Lots of activities can only be done once a day, if you build something it will take at least until the next day for it to show up. The shop refreshes items once a day. 'Collectibles' either randomly show up, at particular times of day, in particular months, or during particular events (easter, etc.).
Furthermore, this 'do nothing and relax' philosophy extends to pretty much everything, so for example buying stuff forces you to go through dialogs every time, and both buying and crafting can't be done in bulk (for the most part). Quite frustrating at times, but I do think it helps me at least not find clever ways to stress myself out and 'optimize' again. I have other games for that :).
Overall, AC is very free from sources of stress, while SV has them everywhere, though it depends on yourself on much they impact you. Also, you can also get stress in AC, but only by setting yourself goals, not through the game itself.