It's fairly easy to cache folders in Github actions, we use it for node_modules, the refreshed when the package.json file changes.

See [1] actions/cache for more details.

[1] https://github.com/actions/cache