If another user/process tries to access your SSH files directly, it'll go through the traditional file permissions to determine if it has access or not. If the disk block is in the page cache AND access is allowed to that inode, then the kernel will retrieve the page from the cache and give it to the process.

To read the whole page cache, you'd need code sitting in kernel space. If something manages to load itself in the kernel space (e.g. kernel module), you have bigger problems to worry about.

I'd be surprised if you can get access to the page/disk cache without root privileges though.

I'm gonna do some more digging on that.

Edit: Maybe Meltdown, lets you access the RAM with that though I guess?