Rebuilding is expensive. We have several caching mechanisms but there is no silver bullet and caching itself can lead to security problems.

For a deeper review of security problems we're currently focussed on please see https://gitlab.com/gitlab-org/gitlab-ce/issues/21911 and please feel free to contribute there.