I think Gitlab recommends a 4GB of memory and suspect this helps with caching and thus page loads. In an organizational context it should not be too hard to throw hardware at the problem. In my organization we run a Gitlab instance with at least 20 active users (70 users total) and quite a lot of repositories and usually it runs quite fast. This is on 8GB of memory and 2 cores.

For personal and home use Gitea/Gogs is probably a better choice indeed. As you point out it runs with much fewer resources and as an added bonus it runs as a single go binary which simplifies hosting it tremendously. You could literally just download the binary and run "$ ./gitea web". Gitlab is another beast in that respect, especially if you do not opt for their Omnibus install.