Task Manager does not report memory use by the OS I/O cache, but there's no general way for it to know what portion of process memory is "necessary" versus "nice to have," so many processes have some type of internal caching (web browsers for example!) that they are able to dump but will be included in their reported memory consumption. Windows has several different layers of caching which makes this a bit complicated to generalize about, but that's sort of the point... Task Manager is not clairvoyant, and so if a process has allocated memory other than specifically through an OS caching infrastructure it reports it as memory in use. The OS caching infrastructure is not so general that it covers every need to hold some memory for performance optimization.
Windows has an infrastructure to prioritize memory availability by process, and to notify processes when there is physical memory pressure so that they can act accordingly. I'm not sure, but as a first-party component I would assume that Windows Defender uses these appropriately.
That said, like most real-time antivirus Defender does feel that it is important to complete real-time protection scans and will sometimes do so at the cost of performance. The logic here is that it is important to complete these scans even under conditions of resource pressure, otherwise malware could just do things like cause high system load before downloading a second stage in order to avoid Defender completing a real-time scan.
Unfortunately this does sometimes cause headaches, for example I saw a situation recently where someone ran a tool that opened a huge number of media files on a NAS in order to read their metadata. This resulted in Defender queuing up a real-time scan of probably over a TB over the network since it saw all of these 10GB+ files being touched, with a definite negative impact on performance. I still wouldn't give "exclude network mounts" as general advice as some people do, but that's an example of sort of a pathological case for real-time scanning where you probably want to exclude it.