The base memory usage will be determined by the kernel modules loaded and the size in memory of those modules. The modules loaded will be based on the enumeration of your hardware. As kernels evolve, the hardware detected evolves and the memory allocated to support the hardware evolves too. Older kernels may have had simpler modules for your specific hardware or in some cases may not have initialized all the hardware. You can compare the output of lsmod on each kernel to compare and contrast memory usage. Other facets of memory allocation evolve over time as well, including buffer memory for things like network stack, socket management, firewall state table and related helper modules, reserved memory for paging, memory mapping of swap and more than I could list on HN. The same kernel you are using right now will have vastly different memory usage on physical bare metal vs. VM's and will vary by what devices are enumerated. Some of the older kernels (1.x) didn't even have dynamically loaded module support and may have only supported a limited set of your hardware.

The file size of the kernel will get bigger over time as more drivers are added for newer hardware. Only the drivers required for your hardware get loaded and thus the memory usage is independent of the kernels file size. If you custom build your kernel, you can strip out all of the drivers not required for your hardware if you had a need to do so. A legit need would be IoT devices that have limited persistent storage. Another use case would be if you were bored and wanted a learning exercise on building kernels or wanted to harden your kernel against specific threats or to add support for hardware without requiring a dynamically loaded module.

If you would like to get back towards a lower usage, some things you could potentially tinker with would be blacklisting drivers you do not need loaded and adjusting sysctl settings, pam limits, systemd unit file limits (does not apply to your 2.4 kernel) for more conservative memory limits based on your needs. Adjusting these things too low can cause problems or performance issues. Use care when blacklisting modules, as some modules depend on other modules and it is not always obvious why. The output of lsmod will show you some examples of the dependencies.