Re: [kata-dev] [Qemu-devel] [PATCH] migration: add capability to bypass the shared memory
On Tue, Jul 3, 2018 at 6:05 PM, Stefan Hajnoczi stefanha@gmail.com wrote:
On Mon, Jul 02, 2018 at 09:52:08PM +0800, Peng Tao wrote:
On Mon, Jul 2, 2018 at 9:10 PM, Stefan Hajnoczi stefanha@gmail.com wrote:
On Sat, Mar 31, 2018 at 04:45:00PM +0800, Lai Jiangshan wrote: Risks:
- If one cloned VM is exploited then all other VMs are more likely to be exploitable (e.g. kernel address space layout randomization).
w.r.t. KASLR, any memory duplication technology would expose it. I remember there are CVEs (e.g., CVE-2015-2877) specific to this kind attack against KSM and it was stated that "Basically if you care about this attack vector, disable deduplication.". Share-until-written approaches for memory conservation among mutually untrusting tenants are inherently detectable for information disclosure, and can be classified as potentially misunderstood behaviors rather than vulnerabilities. [1]
I think the same applies to vm templating as well. Actually VM templating is more useful (than KSM) in this regard since we can create a template for each trusted tenant where as with KSM all VMs on a host are treated equally.
That solves the problem between untrusted users but a breach in one clone may reveal secrets of all other clones belonging to the same tenant. As a user, I would be uncomfortable knowing that if one of my machines is breached then secrets used by all of my machines might be exposed.
Secrets are really point 2 in your list and I'll answer it below :)
- If you give VMs cloned from the same template to untrusted users, they may be able to determine the secrets other users' VMs.
In kata and runv, vm templating is used carefully so that we do not use or save any secret keys before creating the template VM. IOW, the feature is not supposed to be used generally to create any template VMs at any stage.
At what point are templates captured to avoid these problems? Is there code that shows how to do this?
Both runv and kata pauses the VM right after the agent inside guest is up and running, which, in the initramfs case, translates into the point that kernel boots and the init process starts. If you are interested in seeing the actual code, you can look at https://github.com/hyperhq/hyperstart/ and https://github.com/kata-containers/agent for what is done in the guest at that point. If you see any secrets being saved there, I'll be more than happy to fix it. :)
Security is a major factor for using Kata, so it's important not to leak secrets between cloned VMs.
Yes, indeed! And it is all about trade-offs, VM templating or KSM. If we want security above anything, we should just disable all the sharing. But there is actually no ceiling (think about physical isolation!). So it's more about trade-offs. With Kata, VM templating and KSM give users options to achieve better performance and lower memory footprint with little sacrifice. The security advantage of running VM-based containers is still there.
Adding options to enable/disable features leads to confusion among users, makes performance comparisons harder, and increases support overhead.
Technical solutions to the security problems are possible. I'm interested in progress in this area because it means users don't need to make a choice, they can benefit from the feature without sacrificing security.
Well, that is really beyond the scope of the reviewing of this particular QEMU patch. But as a Kata developer, I can answer it anyway.
For one thing, Kata already has quite a few configuration options that let users choose different features. For another thing, Kata already ships with KSM support by default and VM templating in Kata is better off than KSM in many aspects (e.g., by providing similar level of memory conservation w/o affecting host and guest performance). So from Kata containers point of view, it makes sense to have VM templating support and let users decide which one they want to use.
PS: CCing kata-dev since the discussion starts to concern about Kata specific usage.
Cheers, Tao
On Tue, Jul 03, 2018 at 11:10:12PM +0800, Peng Tao wrote:
On Tue, Jul 3, 2018 at 6:05 PM, Stefan Hajnoczi stefanha@gmail.com wrote:
On Mon, Jul 02, 2018 at 09:52:08PM +0800, Peng Tao wrote:
On Mon, Jul 2, 2018 at 9:10 PM, Stefan Hajnoczi stefanha@gmail.com wrote:
On Sat, Mar 31, 2018 at 04:45:00PM +0800, Lai Jiangshan wrote: Risks:
- If one cloned VM is exploited then all other VMs are more likely to be exploitable (e.g. kernel address space layout randomization).
w.r.t. KASLR, any memory duplication technology would expose it. I remember there are CVEs (e.g., CVE-2015-2877) specific to this kind attack against KSM and it was stated that "Basically if you care about this attack vector, disable deduplication.". Share-until-written approaches for memory conservation among mutually untrusting tenants are inherently detectable for information disclosure, and can be classified as potentially misunderstood behaviors rather than vulnerabilities. [1]
I think the same applies to vm templating as well. Actually VM templating is more useful (than KSM) in this regard since we can create a template for each trusted tenant where as with KSM all VMs on a host are treated equally.
That solves the problem between untrusted users but a breach in one clone may reveal secrets of all other clones belonging to the same tenant. As a user, I would be uncomfortable knowing that if one of my machines is breached then secrets used by all of my machines might be exposed.
Secrets are really point 2 in your list and I'll answer it below :)
- If you give VMs cloned from the same template to untrusted users, they may be able to determine the secrets other users' VMs.
In kata and runv, vm templating is used carefully so that we do not use or save any secret keys before creating the template VM. IOW, the feature is not supposed to be used generally to create any template VMs at any stage.
At what point are templates captured to avoid these problems? Is there code that shows how to do this?
Both runv and kata pauses the VM right after the agent inside guest is up and running, which, in the initramfs case, translates into the point that kernel boots and the init process starts. If you are interested in seeing the actual code, you can look at https://github.com/hyperhq/hyperstart/ and https://github.com/kata-containers/agent for what is done in the guest at that point. If you see any secrets being saved there, I'll be more than happy to fix it. :)
Two things come to mind:
At that point both guest kernel and agent address-space layout randomization (ASLR) is finished. ALSR makes it harder for memory corruption bugs to lead to real exploits because the attacker does not know the full memory layout of the process. Cloned VMs will not benefit from ASLR because much of the memory layout of the guest kernel and agent will be identical across all clones.
Software random number generators have probably been initialized at this point. This doesn't mean that all cloned VMs will produce the same sequence of random numbers since they should incorporate entropy sources or use hardware random number generators, but the quality of random numbers might be reduced. Someone who knows random number generators should take a look at this.
Security is a major factor for using Kata, so it's important not to leak secrets between cloned VMs.
Yes, indeed! And it is all about trade-offs, VM templating or KSM. If we want security above anything, we should just disable all the sharing. But there is actually no ceiling (think about physical isolation!). So it's more about trade-offs. With Kata, VM templating and KSM give users options to achieve better performance and lower memory footprint with little sacrifice. The security advantage of running VM-based containers is still there.
Adding options to enable/disable features leads to confusion among users, makes performance comparisons harder, and increases support overhead.
Technical solutions to the security problems are possible. I'm interested in progress in this area because it means users don't need to make a choice, they can benefit from the feature without sacrificing security.
Well, that is really beyond the scope of the reviewing of this particular QEMU patch. But as a Kata developer, I can answer it anyway.
For one thing, Kata already has quite a few configuration options that let users choose different features. For another thing, Kata already ships with KSM support by default and VM templating in Kata is better off than KSM in many aspects (e.g., by providing similar level of memory conservation w/o affecting host and guest performance). So from Kata containers point of view, it makes sense to have VM templating support and let users decide which one they want to use.
PS: CCing kata-dev since the discussion starts to concern about Kata specific usage.
Thanks for doing that. I think discussing the security implications of vm templates (clones) is important specifically for Kata.
Stefan
Hi Stefan,
On Tue, Jul 10, 2018 at 9:40 PM, Stefan Hajnoczi stefanha@gmail.com wrote:
Two things come to mind:
At that point both guest kernel and agent address-space layout randomization (ASLR) is finished. ALSR makes it harder for memory corruption bugs to lead to real exploits because the attacker does not know the full memory layout of the process. Cloned VMs will not benefit from ASLR because much of the memory layout of the guest kernel and agent will be identical across all clones.
Yes, indeed. I am not arguing that ASLR is retained with VM templating. Just that ASLR is also compromised if one wants to use KSM to save memory by sharing among different guests. Kata is already shipping with KSM components and we are adding VM templating as a better alternative.
Software random number generators have probably been initialized at this point. This doesn't mean that all cloned VMs will produce the same sequence of random numbers since they should incorporate entropy sources or use hardware random number generators, but the quality of random numbers might be reduced. Someone who knows random number generators should take a look at this.
As Andrea pointed out earlier in his comments, we can configure the random number generator to printk a warning if it's being used at boot before it had its "shutdown" state restored. Then we can add a new kata-agent request set the entropy and check for such warning after a new VM is cloned and before it is given to the user. This way, we are guaranteed that random numbers generated by each guest is created with a different seed. Do you have other concern with this method?
Cheers, Tao
On Thu, Jul 12, 2018 at 11:02:08PM +0800, Peng Tao wrote:
On Tue, Jul 10, 2018 at 9:40 PM, Stefan Hajnoczi stefanha@gmail.com wrote:
Two things come to mind:
At that point both guest kernel and agent address-space layout randomization (ASLR) is finished. ALSR makes it harder for memory corruption bugs to lead to real exploits because the attacker does not know the full memory layout of the process. Cloned VMs will not benefit from ASLR because much of the memory layout of the guest kernel and agent will be identical across all clones.
Yes, indeed. I am not arguing that ASLR is retained with VM templating. Just that ASLR is also compromised if one wants to use KSM to save memory by sharing among different guests. Kata is already shipping with KSM components and we are adding VM templating as a better alternative.
Hang on, ASLR is *not* compromised by KSM. The address space layout is still unique for each guest, even if KSM deduplicates physical pages on the host. Remember ASLR is about virtual addresses while KSM is about sharing the physical pages. Therefore KSM does not affect ASLR.
The KSM issue you referred to earlier is a timing side-channel attack. Being vulnerable to timing side-channel attacks through KSM does not reduce the effectiveness of ASLR.
Software random number generators have probably been initialized at this point. This doesn't mean that all cloned VMs will produce the same sequence of random numbers since they should incorporate entropy sources or use hardware random number generators, but the quality of random numbers might be reduced. Someone who knows random number generators should take a look at this.
As Andrea pointed out earlier in his comments, we can configure the random number generator to printk a warning if it's being used at boot before it had its "shutdown" state restored. Then we can add a new kata-agent request set the entropy and check for such warning after a new VM is cloned and before it is given to the user. This way, we are guaranteed that random numbers generated by each guest is created with a different seed. Do you have other concern with this method?
Sounds good.
Stefan
participants (2)
-
Peng Tao
-
Stefan Hajnoczi