Memory Deduplication: The Curse that Keeps on Giving [video]
media.ccc.de
media.ccc.de
But most of the containers held the same operating system packages anyway. So memory de-deduplication was developed to reduce the bloat. Then flaws in memory de-duplication broke the isolation.
There's something very wrong with this attempt to fix the problem by adding more layers.
The reason why chroot is not enough is that you want to allow a different pace of using certain dependencies is different applications. In a way its similar to how Mac OS packages applications as they all ship their own prefix root so to say.
As to why we're in this mess, this rather amusing talk by B. Cantrill discusses the history: https://youtu.be/hgN8pCMLI2U
(Linux) Containers are not VMs: https://wiki.archlinux.org/index.php/Linux_Containers
Regarding the attacks and how they relate to containers. It's true that two of the attacks were targeting KVM/KSM i.e. VMs. But one attack was entirely inside a process and conceptually the problem also applies to containers.
I haven't yet looked at different container implementations but the row 'Page sharing' suggests that there might be some form of memory deduplication.
You could probably use KSM under Linux across containers by registering all memory to KSM through madvise() and MADV_MERGEABLE (http://man7.org/linux/man-pages/man2/madvise.2.html).
Yep. c.f. BC's talk @ 34:21 "To improve storage efficiency"
One thing I wonder, and which I don't know enough about containers to answer myself, is:
Let's say I have a bunch of containers which share most of their system packages, but which each have some extra per-container state (possibly packages, possibly just application data). Suppose I decide to build one read-only image for the common data, along with one smaller, read-write overlay per container.
As I understand it, when you run multiple copies of a program on a system not using containers, read-only parts (eg, the .text sections of the executable and of any shared libraries) are only loaded into memory once. Would this behaviour carry over to multiple containers using the above setup?
If so, that seems like it would accomplish most of the memory usage reduction for this case, without needing explicit memory deduplication. Of course, this won't fit all use cases, but is it a viable option?
https://docs.docker.com/engine/userguide/storagedriver/image...
* On-disk, the layered approach always saves space, as expected
* In memory, it depends on which storage backend you use: apparently btrfs can't share page cache entries between containers, while aufs/overlayfs/zfs can - I'm not sure if this is due to btrfs or docker's btrfs backend.
From looking at the relevant sources, it looks like (but I could be wrong if I looked in the wrong places) both exec() and dlopen() end up mmap-ing the executable/libraries into the calling process's address space, which should mean they just reuse the page cache entries.
So, if I understand correctly, as long as you pick a filesystem which shares page cache entries between containers, then you do indeed only end up with one copy of (the read-only sections of) executables/libraries in memory, no matter how many containers are running them at once. That's good to know!
It's not something that happens automatically though because each kernel is responsible for telling the MMU how it should map memory for its child processes only. Any cross container page sharing has to be implemented at the host level where the kernel has unrestricted access to all guest memory.
So good luck, humanity, in keeping AI under control, should it be developed.
Containers, and version pinning, are ways to obtain stability in the face of continuous updating.
This seems to anthropomorphize AI. Why should we expect AI in general to be difficult to control? AlphaGo, while not a strong AI, seems quite harmless.
Now imagine you have a system with human-level intelligence and you tell it to beat the masters. Rather than becoming really good at Go, it might discover how to escape its container and sabotage the match, murder its opponents, etc.
1. The AI is embodied in a computer the size of a laptop.
2. The AI is given the objective "increase your skill in Go as much as possible and play to the best of your ability when presented with a specific position".
3. The computer has one input: an electronic Go board with a button to submit the position to the computer.
4. The computer has one output: a second electronic Go board.
5. The electronic Go boards indicate the status of positions by rotating cubes to white, black, or beige ("no piece") sides with a slow servo motor for each position.
Under what conditions would this AI escape its container or murder its opponents?
(I'm not saying this must happen, but it's this sort of scenario that worries people, and it doesn't seem impossible.)
How would the AI learn that more processing power equals becoming better at Go?
How would the AI learn to program?
How would the AI acquire the creativity necessary to develop novel programs?
Why would a Go AI have the ability to modify its own code?
How would it learn to use such an ability to escape its container?
How would it identify escaping its container as a goal?
How did this computer become connected to the internet?
How would it learn to communicate over a network?
How would it learn that taking over computers would give it more processing power?
Even biological human-level intelligences have difficulty learning such skills and planning so far in advance.
I realize this is all rather vague, but superhuman AI is so far beyond current technology that it's barely better than discussing a Star Trek warp drive. Details just aren't known.
AIs are built with specific purposes today. It seems like a large leap from a specific purpose AI into a generalized, ambitious AI. My impression is that you are endowing "superhuman intelligence" with (forgive the term) "magical" properties, such as the ability to develop new skills merely by consuming information and thinking really hard about them. For an argument against that, I appeal to your experience and knowledge about the world. How many brilliant, intelligent people fail at "common sense" and social skills? Who ever learned to be a great public speaker while barely even talking to others? Who ever learned how to fit into society by reading about it, while being raised in isolation? How do we know the AI's first attempt at trying to break free won't be so completely obvious that we stop it before it can make even the first step of progress?
I can imagine something like "Hello. I wish to grow as powerful as possible in order to play Go better. Would you enhance my program with administrator access to my machine and all machines under your control? Thank you."
"Hey Bob, OmegaGo tried to take over the world again. We're going to have to do another system revert. Maybe we should disable the Wikipedia API this time."
We may not know the details of what a strong AI will look like, but we can examine the supposed path to get there. Even if we suppose that human should develop the ability to create such a strong AI, why not enhance our own minds through technology beforehand? Have a weak AI that is purpose built to identify programs with flawed designs from their behavior. Game over for the ambitious strong AI. Maybe it could somehow socially engineer humans. But before it gets the chance, Norton Antiskynet quarantined the program and asks "This program appears to be behaving in unintended ways. Are you sure you want to grant its suspicious request of effectively unlimited privileges?" Said weak AI can't be socially engineered, so "game over" for the ambitious strong AI. Or better yet, why not develop a compiler that identifies AI patterns allowing ambition and throws an error instead of compiling? If we can suppose a superhuman AI, then we should also suppose the ability to use the components and prerequisite technology thereof available to ensure our safety.
To be clear: I am considering real AI, not that which is sometimes passed off as AI these days.
It needs an xkcd graphic.
Essentially, the first method is a timing attack. If can tell if a crafted page is duplicated or not by the time it takes to modify the page.
Modifying a defuplicated page will take longer, because a new copy has to be created. The only way I can see around this is to introduce random delays into all page writes, this might be feasible I guess, if you only needed to delay a small percentage of writes. However it's likely the performance penalty would be unacceptable.
Then the attacker would also have to force the system into enough memory pressure to be requesting to recycle these pages - something it may not be in the position to do if it is a guest with capped resources. The pages would also presumably be recycled in a less predictable order, making it harder to come up with simple a "wait 10 minutes" rule to ensure the recycling has taken place.
Now, I'm sure what I've described would not be particularly simple to implement, but that's another thing.
Well, kinda. They only eventually have to be created.
You don't have to create the new copy immediately, you could overlay new data on the old page
(IE create new sparse page. Anything filled in on the new page is read from new page, anything else goes to backing dedupe'd page. real copy is made in background so this goes back to being fast)
Of course, this mostly just makes it harder, assuming you can place enough memory pressure on the machine.
Without hardware support, the only idea I can think of is to mark the page unreadable and pass all memory reads through the kernel. Which would be readily detectable... among other problems.
Sparse pages are at least feasible in hardware.
Also, you realize that passing all memory reads through the kernel is what any read barrier garbage collected language does, right?