You might be surprised to hear that your networking card, the AHCI-SATA interface and even the sound-card soldered onto your computer's mainboard is allowed to, and does, read from and write to your computers memory whenever you are using it. {that's just where I had to dig in, in the past, almost any modern peripheral uses DMA}
If the operating system needs a block from your harddisk to be read, and stored in a certain memory location it will actually tell the physical memory location to the AHCI controller and tell it to write the contents into memory, on it's own. It will signal an interrupt when it has finished transfering the data.
Your networking card will have "ring descriptors" which tell the card a list of buffers (in RAM) into which to save new incoming packets. It will raise an interrupt after it has finished writing to the first buffer.
Your sound-card will likewise have a list of buffers from which it reads, and to which it writes just as you are listening to mp3s, or having a phone conversation with google hangouts.
The worst offender would probably be Firewire which, if not configured to block this (modern OSes do), allows these reads from/writes to memory triggered by any connected external device, and from/to addresses supplied by this external device. It can be a huge help for debugging, though.
On typical computers, there's no safeguard against this happening. But some have a IOMMU which can be used to limit DMA and redirect DMA accesses with respect to physical memory.